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27th  Annual  National  Test  and  Evaluation  Conference 

“Test  &  Evaluation:  Serving  the  Warfighter  ” 

Tampa,  FL 

14  -  17  March  2011 


Agenda 


MONDAY.  MARCH  14.  2011 

TUTORIAL  C  -  SESSION  I 

•  11653  -  Test  Planning  —  Advancing  the  Science,  Mr.  Steve  Scukanec,  Senior  Test  Engineer,  Northrop  Grumman  Aerospace  Sector 

•  11678  -  Using  DFSS  as  an  Integrating  Framework  for  MBT&E  and  DOT&E,  Dr.  Mark  Kiemele,  President  and  Cofounder,  Air  Academy 
Associates 

TUTORIAL  G  -  SESSION  I 

•  1 1570  -  A  Day  in  the  Life  of  a  Verification  Statement  Mr.  Steve  Scukanec,  Senior  Test  Engineer,  Northrop  Grumman  Aerospace 

Sector 


TUESDAY.  MARCH  15.  2011 


CONFERENCE  KEYNOTE  ADDRESS 

•  Honorable  Dr.  J.  Michael  Gilmore,  Director,  Operational  Test  &  Evaluation,  OSD 

HOMELAND  SECURITY  T&E  PERSPECTIVES 

•  Mr.  Gary  Carter,  Director,  Test  &  Evaluation  and  Standards  Division,  Department  of  Homeland  Security 

SESSION  B:  OTA’S  (OPERATIONAL  TEST  AGENCY’S)  ROUNDTABLE 

Session  B  Chair  and  Roundtable  Moderator:  Dr.  Catherine  Warner,  Science  Advisor,  DOT&E,  OSD 

SESSION  C:  ACQUISITION  REFORM  -  THE  IMPACT  ON  INDUSTRY 

PENTAGON  RESPONSE  TO  CONGRESSIONAL  STRENGTHENING  OF  DT&E 

•  Mr.  Chris  DiPetto,  Principal  Deputy,  Developmental  Test  &  Evaluation 

REPORT  ON  NDIA’S  INDUSTRIAL  COMMITTEE  ON  TEST  &  EVALUATION  (ICOTE) 

•  Mr.  James  Ruma,  Chairman,  NDIA  ICOTE;  Vice  President,  Engineering,  GDLS 

CONCURRENT  SESSIONS  D  -  K 

•  11627  -  Assessing  System  Reliability  Growth  When  Failure  Modes  are  Masked,  Dr.  Patricia  Jacobs,  Naval  Postgraduate  School 

•  11650  -  Realistic  and  Measurable  Suitability  Requirements  for  Test,  1st  Lt  Andrew  Passey,  USAF,  Air  Force  T&E  Center,  Detachment  6 

•  11563  -  Integrated  Test  and  Independent  Evaluation  (IT&IE)  and  T&E  Using  Experimental  Design  Methodology,  Mr.  George  Axiotis, 
DDR&E/DDT&E 

•  11665  -  OSD  Perspective  of  DT&E  in  Navy  Shipbuilding  Programs,  Mr.  Patrick  Clancy,  OUSD(AT&L)  DDR&E/DDT&E 
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•  11656  -  An  Industry  Response  to  the  Acquisition  Changes,  Mr.  Steve  Scukanec,  Northrop  Grumman  Aerospace  Sector 

•  1 1499  -  Emerging  Methodology  for  Mission-Based  Capability  Assessments,  Mr.  William  Landis,  ARL/SLAD 

•  11557  -  Measures  Development  Standard  Operating  Procedure  (SOP),  Mr.  John  Smith,  Operational  Test  &  Evaluation  Force 

•  11666  -  Understand  the  Mission  -  A  “How-To”  Guide  for  MBTE  Practitioners,  Mr.  Britt  Bray,  DRC 

•  11662  -  Design  Methodology  for  Expedient,  Low  Cost  UAV  Runways,  Mr.  Lorenz  Eber,  Naval  Surface  Warfare  Center,  Dahlgren 

•  12878  -  DoD  Strategic  Planning  for  Test  and  Evaluation,  Mr.  Lee  Schonenberg,  Whitney,  Bradley  and  Brown  Consulting 

•  11709  -  Decoupled  Test,  Evaluation,  and  Certification  of  a  System  of  Systems,  Mr.  Robin  Murray,  JITC 

•  11564  -  The  CRUS  High  Accuracy  TSPI  Architecture  and  Technical  Maturity  Demonstration  Test  Results,  Dr.  Sultan  Mahmood,  Air  Armament 
Center,  AAC/EB 

•  End-to-End  GPS  Multi-Platform  Integrated  System  Testing  for  MGUE,  Dr.  Sultan  Mahmood,  Air  Armament  Center,  AAC/EB 

•  11640  -  Directed  Energy  Test  Tri-Service  Study  2011:  Identifying  Directed  Energy  Test  &  Evaluation  Infrastructure  Requirements,  Mr.  Doug 
Weatherford,  PM  ITTS  IMO 

•  11645  -  Holographic  Radar  Brings  a  New  Dimension  to  Sensing  and  Instrumentation  on  T&E  Ranges,  Mr.  Gary  Kemp,  Cambridge  Consultants 

•  1 1467  -  Guiding  the  Engineer  Through  the  T&E  Process,  Mr.  Allen  Brailey,  Raytheon  Company 

•  1 1483  -  How  to  Frame  a  Robust  Sweet  Spot  Via  Response  Surface  Methods  (RSM),  Mr.  Mark  Anderson,  Stat-Ease,  Inc. 

•  11553  -  MIL-PRF-XX613  and  MIL-STD-X618:  The  Navy  Gets  Serious  About  Armor,  Mr.  Christopher  Brown,  Naval  Surface  Warfare  Center, 
Crane 

•  1 1541  -  Fragment  Analysis  for  the  Joint  Trauma  Analysis  and  Prevention  of  Injury  in  Combat  (JTAPIC),  Ms.  Karen  Pizzolato,  U.S.  Army 
Research  Laboratory 

•  1 1516  -  Mission-Based  Test  and  Evaluation  Strategy:  Progress  Towards  Uniting  Combat  Developer,  Materiel  Developer  and  T&E,  Mr. 
Christopher  Wilcox,  U.S.  Army  Evaluation  Center 

•  11552  -  Using  Complementary  Frameworks  for  Qualitative  Data  Collection  During  OT&E:  Piggybacking  on  Operational  Experiments,  Ms. 
Chiesha  M.  Stevens,  Pacific  Science  &  Engineering  Group,  Inc. 

•  11699  -  Continuous  Cost  Reduction  Feeds  Back  into  Product  Reliability,  Mr.  Jonathan  Nikkei,  Raytheon  Missile  Systems 

•  11704  -  Testing  &  Evaluating  the  Net-  Ready  Key  Performance  Parameter  (KPP),  Ms.  Danielle  Koester,  JITC 


WEDNESDAY.  MARCH  16.  2011 


SESSION  L:  A  RE-ENERGIZED  DT&E 

PANEL:  T&E:  SERVING  THE  WARFIGHTER  IN  A  COST-CONSTRAINED  ENVIRONMENT 
Panel  Moderator: 

•  Mr.  Chris  DiPetto,  Principal  Deputy,  Developmental  Test  &  Evaluation 

Panelists: 

•  Mr.  David  K.  Grimm,  Acting  Director,  Deputy  Under  Secretary  of  the  Army,  T&E  Office 

•  Mr.  Steve  Hutchison,  DISA  T&E  Executive 

SESSION  M:  RESPONSIVE  AND  AGILE  INFORMATION  SYSTEMS  T&E  PANEL 
Session  M  Chair  and  Panel  Moderator: 

•  Dr.  Steve  Kimmel,  Chairman,  NDIA  C4ISR  Division;  Senior  Vice  President,  Alion  Science  &  Technology 

SESSION  N:  IMPROVING  THE  T&E  PROCESS 

SOCOM  T&E  PERSPECTIVES:  SERVING  THE  WARFIGHTER 

•  LTC  Kevin  Vanyo,  USA,  USSOCOM  J8-0 

CONCURRENT  SESSIONS  O  -  V 

•  11560  -  A  Comprehensive  Approach  to  Characterizing  the  Hazards  of  Explosive  Countermeasures  With  Respect  to  Dismounted  Troops,  Mr. 
Stephen  Swann,  U.S.  Army  Research  Laboratory 

•  11529  -  Expanding  Use  of  the  Probability  of  Raid  Annihilation  (PRA)  Test  Bed  Across  the  Ship  Self-Defense  Enterprise,  Mr.  Richard 
Lawrence,  AVW  Technologies 

•  11500  -  Modeling  and  Simulation  for  Mission-Based  Test  and  Evaluation  (MBT&E),  Mrs.  Beth  Ward,  U.S.  Army  Research  Laboratory 

•  1 1476  -  A  Paradigm  for  Modeling  and  Simulation  in  support  of  Mission-Based  Test  and  Evaluation,  Dr.  James  Walbert,  SURVICE  Engineering 
Company 

•  1 1497  -  Joint  Mission  Environment  Test  Capability  (JMETC):  Improving  Distributed  Capabilities,  Mr.  Chip  Ferguson,  JMETC 

•  1 1 508  -  U.S.N.  RDTE  Project  Support  Aircraft,  Mr.  Charles  Myers,  U.S.  Navy,  NAWCAD 

•  11626  -  Dugway  Proving  Ground  as  the  MRTFB  Chem  Bio  Activity,  Ms.  Jean  Baker,  U.S.  Army  Dugway  Proving  Ground 

•  11677  -  Using  Design  of  Experiments  (DoE)  to  Integrate  Developmental  and  Operational  T&E,  Dr.  Mark  Kiemele,  Air  Academy  Associates 

•  11549  -  Probability  Driven  Experiments  Design  for  Autonomous  Systems,  Mr.  Troy  Jones,  Charles  Stark  Draper  Laboratory 

•  11532  -  Design  of  Experiments:  Managing  Expectations,  Mr.  James  Carpenter,  AVW  Technologies,  Inc. 

•  11538  -  Personnel  Injury  Analysis  of  Reflective  Spall,  Mrs.  Rebecca  VanAmburg,  U.S.  Army  Research  Laboratory 

•  11539  -  Analytical  Approach  Using  MUVES-S2/ORCA  Modeling  in  Support  of  the  Joint  Cargo  Aircraft  (JCA),  Mr.  Richard  Moyers,  U.S. 
Army  Research  Laboratory 
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•  11674  -  Utilization  of  Model  and  Simulation  for  Network  Waveform  Characterization  and  Validation,  Mr.  Scott  Rediger,  Rockwell  Collins 

•  11676  -  Model  Based  Systems  Engineering  and  M&S  Adding  Value  to  T&E,  Mr.  Larry  Grello,  High  Performance  Technologies,  Inc. 

•  11554  -  The  Impact  of  High  Accuracy  Target  Geometry  in  Modeling  and  Simulation  to  Support  Live  Fire  Test  and  Evaluation,  Mr.  Scott 

Homung,  U.S.Army  Research  Laboratory/  SLAD 

•  1 1638  -  Army  Testing  in  a  Services  Oriented  Architecture  (SOA)  Environment,  Mr.  Michael  Phillips,  Mantech  International 

•  11639  -  The  Test  and  Training  Enabling  Architecture  (TENA)  Enabling  Technology  for  the  Joint  Mission  Environment  Test  Capability 
(JMETC)  in  Live,  Virtual,  and  Constructive  (LVC)  Environments,  Mr.  Gene  Hudgins,  TENA/JMETC 

•  11682  -  Advanced  Range  Data  System  (ARDS)  Service  Life  Extension  Program  (SLEP)  -  “Ensuring  GPS  Based  TSPI  Remains  a  Viable  T&E 
Range  Instrumentation  Asset”,  Mr.  Dick  Dickson, TYBRIN  Corporation 

•  11698  -  Target  Systems  in  Support  of  Test  and  Evaluation,  Mr.  James  Schwierling,U.S.  Army  Targets  Management  Office 

•  11524  -  Ready  for  Scrum?  Dr.  Steven  Hutchison,  DISA 

•  11649  -  Affordable  Test  and  Evaluation  in  a  Complex  World,  Mr.  Thomas  Wissink,  Lockheed  Martin 

•  1 1710  -  Testing  U.S.  Systems  for  Coalition  Interoperability,  LTC  Tim  Timmons,  USA,JITC 

•  11659  -  Impacts  of  the  Learning  Curve  -  Operational  Test  &  Evaluation,  Ms.  Shannon  Krammes,  MCOTEA 


THURSDAY.  MARCH  17.  2011 

SESSION  W:  TEST  DESIGN,  TEST  CURRICULA  AND  STANDARDS 

•  11690  -  Doing  More  Without  More  -  Scientific  T&E  Design  Methodologies  (STED  in  DOD  Weapons  Systems  Aquisition),  Ms.  Darleen 
Mosser-Kemer,  Deputy  Director,  Capabilities  Development,  Office  of  the  Director,  DT&E 

•  Report  On  Standards  For  DT&E,  CDR  Ernest  Swauger,  USN  (Ret),  JPEO-CBD/Chief,  CM/HD  Systems  IP  AT 

•  11663  -  Effective  Combat  Data  Collection  &  Applicability  to  T&E,  LtCol  Michael  Kennedy,  USMC,  Expeditionary  Test  Division,  MCOTEA 

•  11651  -  Test  &  Evaluation  Issues  For  Systems  of  Systems  (SoS):  Creating  Sleep  Aids  For  Those  Sleepless  Nights,  Dr.  Beth  Wilson,  Principal 
Engineering  Fellow,  Raytheon  Company 

•  11569  -  T&E  -  Guarding  The  Requirements  Intent,  Mr.  Steve  Scukanec,  Senior  Test  Engineer,  Northrop  Grumman  Aerospace  Sector 
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Top  Pentagon  leadership 
presentations  on  T&E  / 
acquisition  policy  and  issues 
Industry  leaders  sharing  T&E 
perspectives  and  responses  to 
recent  policy  initiatives 
Special  former  NSA  guest 
speaker  addressing  cyber 
security  policy 
Over  80  speakers  addressing 
a  host  of  issues  facing  today’s 
T&E  community 
Parallel  breakout  sessions 
focused  on  specific  T&E  issues 


Co-Sponsored  by  the  NDIA  C4ISR  &  Systems 
Engineering  Divisions 


MARRIOTT  TAMPA  WATERSIDE  TAMPA,  FLORIDA 


WWW.NDIA.ORG/MEETINGS/1 91 0 


EVENT  #1910 


TEST  &  EVALUATION  CONFERENCE 
GENERAL  INFORMATION 


CONFERENCE  ANNOUNCEMENT 

The  27th  Annual  National  Test  &  Evaluation  Conference  is  sponsored  by  the  NDIATest 
&  Evaluation  Division  and  supported  by  the  Office  of  the  Under  Secretary  of  Defense 
(AT&T)  and  the  Director,  Operational  Test  &:  Evaluation  (DOT&:E).  Co-sponsors  of  this 
symposium  are  the  C4ISR  and  Systems  Engineering  Divisions  of  NDIA. 

Test  and  Evaluation  is  often  looked  at  by  Program  Managers,  Program  Executive  Officers 
and  other  proponents  of  weapon  systems  as  an  unwelcome  obstacle  to  the  deployment  of 
systems  to  the  Department  of  Defense  and  Homeland  Security.  T&;E  is  often  seen  as  a 
source  of  bad  news  which  can  potentially  delay  the  deployment  of  these  systems  and  add 
to  their  eventual  cost. 

Most  engineers,  technicians  and  program  administrators  recognize  that  test  and  evaluation 
is  an  integral  part  of  the  scientific  method  of  systematically  assessing  the  effectiveness, 
suitability  and  survivability  of  hardware,  software  and  personnel. 

This  national  conference  will  focus  on  policies,  methods,  and  approaches  that  could  better 
serve  the  ultimate  consumer  of  our  T&:E  efforts,  the  Warfighter.  Given  that  Tampa  is  the 
home  of  both  the  U.S.  Special  Operations  Command  and  the  U.S.  Central  Command,  it 
will  provide  a  fertile  opportunity  to  see  and  hear  first-hand  about  how  T&:E  could  better 
serve  our  fighting  forces. 

With  the  recent  combat  surge  into  Afghanistan  and  change  in  our  operational  support  in 
Iraq,  it  is  vital  that  we  take  note  of  the  recent  lessons  learned  in  both  rapid  deployment 
as  well  as  tailoring  our  responses  to  the  changing  environments  and  tactics  our  fighting 
forces  are  now  facing. 

Increasing  fiscal  pressures  also  prompt  us  to  address  T&:E  approaches  to  saving  time  and 
money  as  well  as  to  examine  those  other  disciplines  which  feed  theT&:E  activity,  including 
Systems  Engineering,  Logistics,  C4ISR,  and  R&:D  and  Training. 

Recent  policy  initiatives  will  also  be  addressed  as  to  their  implications,  applications  and 
effectiveness.  Discussions  will  include  how  the  recent  legislative  initiatives  requiring 
additional  T&;E  statutory  responsibilities  for  Developmental  Test  and  Evaluation  are 
being  implemented.  Multiple  topic  tracks  and  tutorial  sessions  will  be  included  in  the 
conference  to  enable  more  focused  discussions  of  specific  topics  enabling  additional  time 
for  Q&A  as  well. 


CONFERENCE  ATTIRE 

Conference  attire  is  business  for  civilians  and  Class  A  uniform  for  military.  In  addition, 
your  identification  badge,  received  upon  conference  check-in,  must  be  worn  at  all  times. 


NDIA  T&E  EXECUTIVE 
BOARD 

Mr.  Joe  Andrese,  AEG  NDIA 
Chapter  * 

Dr.  Suzanne  Beers,  MITRE 
Corporation 

Dr.  Keith  Bradley,  LLNL 

Mr.  Britt  Bray,  DRC 

Corporation 

Mr.  Sam  Campagna,  NDIA 

RADM  David  Crocker,  USN 
(Ret),  Booz  Allen  Hamilton 

Dr.  Paul  Deitz,  AMSAA* 

Mr.  Dick  Dickson,  Tybrin 
Corporation 

Dr.  Anne  Hillegas,  ARA 

Corporation 

Mr.  John  Illgen,  Northrop 
Grumman 

RADM  Bert  Johnston,  USN 

(Ret),  Wyle  Corporation 

Dr.  Mark  Kiemele,  Air  Academy 
Associates 

Mr.  Chuck  Larson,  SURVICE 
Engineering 

Mr.  James  O’Bryon,  The 

OBryon  Group ,  T&E  Division 
Chair 

Mr.  Brendan  Rhatigan, 

Lockheed  Martin 

Mr.  Jack  Sheehan,  ORSA 
Corporation 

Dr.  James  Streilein,  OSD, 
DOT&E* 

Dr.  Lowell  Tonnessen,  IDA 

Dr.  Juan  Vitali,  OSD  CBD * 

Mr.  Martin  Woznica,  Raytheon 
Company 

Mr.  William  Yeakel,  ORSA 
Corporation 

*  Government  liaison  to  NDIA 
T&E  Executive  Board 


TEST  &  EVALUATION  CONFERENCE 
AWARD  INFORMATION 


WALTER  W.  HOLLIS  HONORS  LUNCHEON 

Hie  Walter  W.  Hollis  Award  is  presented  annually  in  recognition  of  lifetime  contributions  and  achievement  in  the  area  of  defense  Test  & 
Evaluation.  The  award  is  presented  in  the  name  of  Walter  W.  Hollis  who  is  recognized  for  his  dedicated  and  long-standing  service  in  the 
field  of  Defense  Test  &  Evaluation.  This  year’s  recipient,  Dr.  James  N.  Walbert,  Chief  Scientist,  SURVICE  Engineering  Company,  will  be 
recognized  at  the  conference  Awards  Luncheon  on  Tuesday,  March  15. 

Previous  Recipients  of  this  Award: 

Dr.  James  J.  Streilein,  Technical  Director/ Deputy  to  the  Commander. ;  U.S.  Army  Test  and  Evaluation  Command  (2010) 

Dr.  Ernest  Seglie,  Science  Advisor  to  the  Director,  Operational  Test  &  Evaluation,  OSD  (2009) 

Dr.  Paul  H.  Deitz,  Technical  Director,  AMSAA,  AEG,  MD  (2008) 

Mr.  James  F.  O’Bryon,  Former  DDOT&E  /  LFT  (2007) 

RADM  Charles  “Bert”  Johnston,  USN  (Ret),  Wyle  Laboratories  (2006) 

Hon  Thomas  Christie,  DOT&E,  OSD  (2005) 

Dr.  Marion  Williams,  HQAFOTEC  (2004) 

Mr.  James  Fasig,  Aberdeen  Test  Center  (2003) 

Mr.  G.  Thomas  Castino,  Underwriters  Laboratories,  Lnc.  (2002) 

Hon  Philip  Coyle,  III,  DOT&E,  OSD  (2001) 

Mr.  Walter  W.  Hollis,  Department  of  the  Army  (2000) 


TESTER  OF  THE  YEAR  AWARDS  LUNCHEON 

These  awards,  presented  to  outstanding  individuals  in  the  field  of  Test  &  Evaluation,  offer  OSD  and  each  Military  Service  Test  &  Evaluation 
Department  the  opportunity  to  select  three  award  recipients  for  recognition  as  the  Tester  of  the  Year  in  specific  categories.  The  three  categories 
recognized  are:  Military,  Civilian,  and  Contractor.  Recipients  will  be  recognized  at  the  conference  Awards  Luncheon  on  Wednesday,  March 
16. 


MAJ  Brian  Spurlock,  USA 

2010 

Army 

Military  Tester  of  the  Year 

Ms.  Patricia  Frounfelker 

2010 

Army 

Civilian  Tester  of  the  Year 

Mr.  Henry  Waller 

2010 

Army 

Contractor  Tester  of  the  Year 

COL  Steven  Duke,  USA 

2010 

OSD 

Military  Tester  of  the  Year 

Ms.  Stephanie  Koch 

2010 

OSD 

Civilian  Tester  of  the  Year 

Mr.  Patrick  Matthews 

2010 

OSD 

Contractor  Tester  of  the  Year 

Maj  Ryan  Voneida,  USAF 

2010 

USAE 

Military  Tester  of  the  Year 

Mr.  William  Nix 

2010 

USAF 

Civilian  Tester  of  the  Year 

Mr.  David  Smith 

2010 

USAE 

Contractor  Tester  of  the  Year 

CDR  John  Verniest,  USN 

2010 

Navy 

Military  Tester  of  the  Year 

Mr.  Don  Nelson 

2010 

Navy 

Civilian  Tester  of  the  Year 

Mr.  Douglas  Cornell 

2010 

Navy 

Contractor  Tester  of  the  Year 

Capt  Todd  Richardson,  USMC 

2010 

Marine  Corps 

Military  Tester  of  the  Year 

Ms.  Cam  Donohue 

2010 

Marine  Corps 

Civilian  Tester  of  the  Year 

Mr.  Eric  Rannenberg 

2010 

Marine  Corps 

Contractor  Tester  of  the  Year 
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MONDAY,  MARCH  14, 2011 


10:00  AM  -  6:00  PM 

CONFERENCE  REGISTRATION  OPEN  -  2ND  LEVEL  REGISTRATION 

10:00  AM  -  2:45  PM 

TUTORIALS  A-D,  SESSION  1  -  SEE  TRACK  LAYOUT  FOR  ROOM  ASSIGNMENTS 

There  is  a  $50  registration  fee  for  tutorial  attendance. 

11 :00  AM -4:00  PM 

DISPLAY  SET-UP  -  GRAND  SALONS  A-D 

12:00  NOON  - 1:00  PM  LUNCH  BREAK 

Lunch  not  included  in  conference  or  tutorial  registration 


2:45  PM  -  3:00  PM 

AFTERNOON  BREAK  -  GRAND  BALLROOM  FOYER 

For  tutorial  registrants  only 

3:00  PM  -  4:30  PM 

TUTORIALS  E-H,  SESSION  2  -  SEE  TRACK  LAYOUT  FOR  ROOM  ASSIGNMENTS 

4:30  PM 

TUTORIALS  CONCLUDE 

5:00  PM  -  6:00  PM 

6:00  PM 

KICKOFF  RECEPTION  IN  THE  DISPLAY  AREA  -  GRAND  SALONS  A-D 

Open  to  all  conference  registrants 

CONFERENCE  ADJOURNED  FOR  THE  DAY 

6:00  PM 


TEST  &  EVALUATION  CONFERENCE 
MONDAY,  MARCH  14,  2011 


MONDAY,  MARCH  14,  2011  —  Tutorials 


10:00  AM -2:45  PM 


TUTORIAL 

TUTORIAL  A 

Session  Chair:  Dr.  Paul  Deitz, 
AMSAA 

Grand  Salon  G 

TUTORIAL  B 

Session  Chair:  Mr.  Martin  Woznica, 
Raytheon  Company 

Grand  Salon  H 

TUTORIAL  C 

Session  Chair:  Dr.  Suzanne  Beers, 
MITRE  Corporation 

Grand  Salon  I 

TUTORIAL  D 

Session  Chair:  Mr.  Britt  Bray,  DRS 
Corporation 

Grand  Salon  J 

SESSION  1 

SESSION  1 

SESSION  1 

SESSION  1 

10:00  AM 

1 1678  -  Using  DFSS  as  an 

Integrating  Framework  for  MBT&E 
and  DOT&E 

Dr.  Mark  Kiemele,  President  and  Co¬ 
founder,  Air  Academy  Associates 

1 1694  -  Efficient  Modeling  and 
Simulation  (M&S)  Using  Design  of 
Experiments  (DOE)  Methods 

Dr.  Tom  Donnelly,  Principal 

Customer  Advocate,  Systems  Engineer, 
JMP 

11653  -  Test  Planning  —  Advancing 
the  Science 

Mr.  Steve  Scukanec,  Senior  Test 
Engineer,  Northrop  Grumman 

Aerospace  Sector 

1 1488  -  Testing  and  Evaluating 
Intranets,  Portals,  and  Enterprise 
Systems  for  Usability 

Dr.  Patricia  Chalmers,  Chief  Science 
Advisor,  U.S.  Joint  Forces  Command 

SESSION  1  CONTINUED 

SESSION  1  CONTINUED 

SESSION  1  CONTINUED 

SESSION  1  CONTINUED 

1:00  PM 

1 1678  -  Using  DFSS  as  an 

Integrating  Framework  for  MBT&E 
and  DOT&E 

Dr.  Mark  Kiemele,  President  and  Co¬ 
founder,  Air  Academy  Associates 

1 1694  -  Efficient  Modeling  and 
Simulation  (M&S)  Using  Design  of 
Experiments  (DOE)  Methods 

Dr.  Tom  Donnelly,  Principal 

Customer  Advocate,  Systems  Engineer, 
JMP 

11653  -  Test  Planning  —  Advancing 
the  Science 

Mr.  Steve  Scukanec,  Senior  Test 
Engineer,  Northrop  Grumman 

Aerospace  Sector 

1 1488  -  Testing  and  Evaluating 
Intranets,  Portals,  and  Enterprise 
Systems  for  Usability 

Dr.  Patricia  Chalmers,  Chief  Science 
Advisor,  U.S.  Joint  Forces  Command 

3:00  PM  -  4:30  PM 


TUTORIAL 

TUTORIAL  E 

Session  Chair:  Dr.  Lowell  Tonnessen, 
IDA 

Grand  Salon  G 

TUTORIAL  F 

Session  Chair:  Mr.  Dick  Dickson, 
Tybrin  Corporation 

Grand  Salon  H 

TUTORIAL  G 

Session  Chair:  Mr.  Chuck  Larson, 
SURVICE  Engineering 

Grand  Salon  I 

TUTORIAL  H 

Session  Chair:  Mr.  Brendon 
Rhatigan,  Lockheed  Martin 

Grand  Salon  J 

SESSION  2 

SESSION  2 

SESSION  2 

SESSION  2 

1 1703  -  Ships  Are  Different 

Mr.  Mark  Lucas,  Command  Technical 

11693  -  Modern  Design  of 
Experiments  (DOE)  Methods 

11570  -  A  Day  in  the  Life  of  a 
Verification  Statement 

11705  -  Defense  Information 

Systems  Agency  Joint 

Interoperability  Test  Command 

3:00  PM 

Director,  Combat  Direction  Systems 
Activity 

Dr.  Tom  Donnelly,  Principal 

Customer  Advocate,  Systems  Engineer, 
JMP 

Mr.  Steve  Scukanec,  Senior  Test 
Engineer,  Northrop  Grumman 

Aerospace  Sector 

Interoperability  Support  for  the 
Afghanistan  Mission  Network 

Mr.  Jeffery  Phipps,  CIAV  Co-Chair, 

US  Lead,  JITC 
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TUTORIAL  A:  USING  DFSS  AS  AN  INTEGRATING 
FRAMEWORK  FOR  MBT&E  AND  DOT&E 

This  tutorial  will  provide  attendees  a  comprehensive  process 
to  capture  all  of  the  activities  in  MBT&E  and  DOT&E 
needed  to  achieve  a  successful  system  acquisition.  It  will  use 
DFSS  in  its  more  expansive  connotation,  namely  Designing 
for  Successful  Systems  vice  Design  for  Six  Sigma,  the  more 
common  but  limited  meaning.  DFSS  starts  with  the  voice 
of  the  warfighter  (or  customer)  and  the  required  operational 
capability.  These  requirements  are  then  flowed  down  to  the 
critical  performance  measures  using  tools  that  help  to  prioritize 
along  the  way.  The  performance  measures  may  include  KPPs, 
MOEs,  MOSs,  and  CTPs.  The  critical  performance  measures 
are  linked  to  key  design  parameters,  and  once  this  linkage  is 
firm,  performance  optimization  can  be  accomplished.  Design 
of  Experiments  (DOE)  is  shown  to  be  a  critical  player  in  the 
design  and  optimization  phases,  as  well  as  in  every  facet  of 
testing  and  evaluation.  Once  the  design  and  performance  is 
optimized,  it  must  be  validated  and  the  capability  rolled  back 
up  to  the  system  level  capability.  DFSS  will  be  shown  as  an 
interdisciplinary  activity,  spanning  the  activities  of  systems 
engineering,  reliability  engineering,  design  and  optimization, 
test  and  evaluation,  and  system  capability  confirmation. 

TUTORIAL  B:  EFFICIENT  MODELING  AND  SIMULATION 
(M&S)  USING  DESIGN  OF  EXPERIMENTS  (DOE) 
METHODS 

Attendees  will  learn  how  Design  of  Experiments  (DOE) 
methods  can  be  used  to  extract  the  most  useful  information 
from  computer  simulation  models.  They  will  see  how  the 
sequential  running  of  blocks  of  simulations  can  be  used  to 
conduct  the  overall  fewest  trials  necessary  to  do  sensitivity 
analysis  of  the  factors  being  studied.  They  will  also  see  how 
to  develop  a  fast-running  (seconds)  surrogate  model  —  which 
testers  and  analysts  can  interactively  query  —  of  a  long- 
running  (hours,  days  or  weeks)  simulation.  Design  solutions 
will  include  the  application  of  traditional  DOE  methods  to 
discrete  event  and  agent-based  simulations,  and  modern  space¬ 
filling  designs  to  more  complex  physics-based  simulations 
such  as  Computational  Fluid  Dynamics  (CFD).  When  to 
use,  and  how  to  choose  between  traditional  linear  regression 
approximation  methods  and  spatial  regression  interpolation 
methods  will  be  discussed.  The  effective  practice  of  using 
checkpoint  simulations  for  determining  the  accuracy  of 
surrogate  model  predictions  will  be  demonstrated. 


TUTORIAL  C:  TEST  PLANNING  —  ADVANCING  THE 
SCIENCE 

Test  planning  is  rapidly  becoming  a  lost  art.  Many  test 
planning  activities  are  based  solely  on  corporate  knowledge  and 
“Like  we  did  it  last  time”  theories.  Solidifying  requirements 
development,  improving  the  programs  verification  and 
validation  activities,  increased  program  collaboration  and 
streamlined  test  programs  are  all  benefits  of  a  solid  and 
well  defined  test  planning  approach.  By  increasing  program 
collaboration  and  the  overall  time  spent  on  the  “engineering 
of  a  program”  while  significantly  reducing  the  time  required 
producing  the  engineering  verification  and  validation  artifacts, 
solid  model  based  test  planning  can  ensure  that  a  test  program 
is  more  effective  across  its  lifecycle.  This  tutorial  examines  the 
test  planning  process.  From  verification  to  test  plan  modeling 
and  test  plan  generation,  participants  will  see  the  processes  and 
tool  sets  in  action.  To  demonstrate  some  of  these  capabilities, 
participants  will  generate  test  requirements  and  objectives, 
model  the  plan,  optimize  the  plan  and  assign  resources, 
and  finally  generate  a  simple  test  plan  while  maintaining 
connections  to  the  original  requirements  intent. 

TUTORIAL  D:  TESTING  AND  EVALUATING  INTRANETS, 
PORTALS,  AND  ENTERPRISE  SYSTEMS  FOR 
USABILITY 

This  tutorial  will  teach  attendees  how  to  perform  intranet, 
portal,  and  enterprise  usability  evaluations.  Attendees  are 
encouraged  to  come  with  a  project  in  mind  as  they  will  be 
worked  on  throughout  the  tutorial.  Attendees  will  learn 
how  to  analyze  their  stakeholders’  goals  and  needs:  How  to 
decide  who  their  stakeholders  are,  decide  which  stakeholders 
to  include  in  their  evaluation,  choose  a  random  sample  of  end 
users,  and  determine  stakeholders’  goals/needs.  Attendees  will 
learn  how  to  design  a  Usability  Evaluation:  How  to  budget 
time,  knowing  what  types  of  T&E  methods  are  possible, 
deciding  what  methods  to  use,  designing  a  first-rate  survey, 
determining  sample  completion  tasks,  deciding  how  many 
methods  to  use,  and  how  to  quantify  usability  data.  Attendees 
will  write  a  design  for  their  portal  evaluation  including  topics 
discussed.  Information  will  be  provided  on  How  to  Evaluate 
Your  Portal  Usability  Evaluation:  Pilot  evaluations,  participant 
performance,  survey  understandability,  task  understandability, 
determining  if  tasks  are  too  easy  or  too  hard,  understanding 
the  data,  feedback  from  participants,  making  improvements. 
Attendees  will  also  learn  how  to  write  their  reports.  Portal 
evaluation  samples  will  be  provided. 
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TUTORIAL  E:  SHIPS  ARE  DIFFERENT 

Recent  fleet  concerns  with  surface  ship  and  system 
performance  have  punctuated  the  need  to  evolve  the 
Navy’s  ship  T&E  processes  and  practices  in  such  a  way  that 
enables  acquisition  decisions  that  are  based  on  a  framework 
of  mission  area  effectiveness  and  suitability.  However, 
because  any  given  ship  supports  multiple  missions  through 
the  employment  of  a  complex  array  of  systems,  sensors, 
and  weapons,  the  aforementioned  changes  truly  require 
a  “system  of  systems”  approach.  This  approach  must  take 
care  in  balancing  multiple  systems  at  differing  states  of 
lifecycle  maturity  through  their  development  processes.  This 
necessitates  a  progressive  examination  of  systems  maturity 
using  mission-based,  measureable,  testable  artifacts.  This 
tutorial  will  discuss  the  Navy’s  Mission  Based  Test  Design 
methodology  and  illustrate  how  its  application  through  an 
Integrated  Test  process  can  be  used  in  ship  and  ship  systems 
acquisition.  It  will  also  discuss  how  this  approach  can  enable 
improved  rigor  leading  to  a  better  understanding  of  risks 
and  warfighting  effects,  thereby  facilitating  the  information 
quality  needed  for  effective  ship  deployment  decisions. 

TUTORIAL  F:  MODERN  DESIGN  OF  EXPERIMENTS 
(DOE)  METHODS 

This  tutorial  will  provide  attendees  the  very  latest 
experimental  designs  published  since  2008.  References  will 
be  provided  for  four  new  types  of  design  that  offer  testers  the 
ability  to  run  either  fewer  trials  or  for  the  same  number  of 
trials,  learn  more  about  interactions  or  quadratic  behavior. 
These  recently  peer-reviewed  designs  have  not  yet  made 
it  into  textbooks.  The  new  designs  include  non-regular 
orthogonal  fractional-factorial,  robust  screening,  alias- 
optimal,  and  Bayesian  D-optimal  supersaturated  designs. 
Comparisons  between  these  new  alternative  methods 
and  traditional  designs  will  be  provided  to  show  the  new 
methods  are  superior  or  strong  competitors. 

TUTORIAL  G:  A  DAY  IN  THE  LIFE  OF  A  VERIFICATION 
STATEMENT 

One  measure  of  the  quality  of  a  product  requirement  is  that  it 
be  verifiable.  Verifiability  assessment  is  one  of  the  exit  criteria 
for  the  Systems  Requirements  Review  and  is  necessary  for 
requirement  validity.  Nomination  of  one  or  more  verification 
methods  (examination,  analysis,  demonstration  or  test)  is 
often  taken  as  the  sole  evidence  of  verifiability.  A  completed 
Verification  Cross  Reference  Matrix  is  frequently  considered 


as  the  final  verifiability  assessment  and  responsibility  for 
the  remainder  of  the  verification  effort  is  transferred  to  the 
test  and  evaluation  and  other  implementing  communities 
for  completion.  Lessons  learned  from  many  programs  have 
shown  that  a  more  robust  application  of  systems  engineering 
should  include  the  requirements  engineers  (with  detailed 
knowledge  of  product  requirement  intent)  working  with 
the  verification  implementing  organizations  as  the  best 
combination  to  define  the  verification  requirements.  Such 
definition  should  include  statement  of  the  verification 
objectives,  success  criteria  and  environment.  Including 
this  information  in  the  ’’Quality  Assurance”  section  of  the 
requirements  document  allows  for  buy-in  by  the  customer 
well  in  advance  of  implementing  the  verification  activities. 
This  information  is  used  by  verification  personnel  to 
generate  one  or  more  verification  plans  and  to  develop  the 
detailed  verification  program.  Verification  requirements 
are  planned  into  verification  events  which  are  executed 
using  the  proper  system  elements  and  environments.  These 
verification  requirements  are  key  to  establishing  long  lead 
verification  facilities,  tools  and  laboratories.  Early  definition 
of  these  requirements  helps  prevent  facility  re-designs  and 
verification  re-plans  that  can  cause  expensive  delays.  Finally, 
verification  data  analysis  is  performed,  and  the  information 
compiled  into  verification  reports  certifying  system  product 
requirements  compliance.  This  robust  verification  approach 
will  provide  proof  of  requirements  satisfaction,  leading  to 
systems  that  meet  the  customers’  needs  at  a  lower  life-cycle 
cost.  This  presentation  explores  the  value  of  well-crafted 
verification  requirements  developed  early  in  the  Program. 
A  “Day  in  the  Life  of  a  Verification  Requirement”  shows 
the  interaction  and  benefits  of  verification  requirements  to 
the  verification  execution  teams.  The  presentation  will  offer 
a  lifecycle  description  of  the  verification  requirement  from 
conception  to  certification. 
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TUTORIAL  H:  DEFENSE  INFORMATION  SYSTEMS  AGENCY  JOINT  INTEROPERABILITY  TEST  COMMAND 
INTEROPERABILITY  SUPPORT  FOR  THE  AFGHANISTAN  MISSION  NETWORK 

USCENTCOM  operates  in  a  coalition  environment  and  must  be  able  to  generate  and  pass  critical  information  to  U.S.  and 
coalition  partners.  The  Command  and  NATO,  as  members  of  the  International  Security  Assistance  Forces  (ISAF),  understand 
that  widespread  interoperability  is  a  key  component  to  achieve  effective  and  efficient  operations.  These  communication 
capabilities  must  include  a  wide  variety  of  not  only  military  governmental  operations,  but  also  non-governmental  agencies 
and  industrial  partners.  To  that  end,  they’ve  created  the  Afghan  Mission  Network  (AMN)  and  commissioned  the  Defense 
Information  Systems  Agency’s  Joint  Interoperability  Test  Command  to  develop  the  Coalition  Test  and  Evaluation  Environment 
(CTE2)  testing  arm  of  the  Coalition  Interoperability  Assurance  and  Validation  (CIAV)  process.  The  AMN  is  the  backbone  or 
core  infrastructure  that  will  provide  long-term  communications  and  information  system  and  satellite  communication  services 
to  support  the  ISAF  as  it  expands  its  operations  across  the  country  during  the  ongoing  operations.  This  tutorial  will  discuss 
the  eight  core  critical  Coalition  Mission  threads,  phases  for  testing,  and  how  the  JITC  stood  up  a  network  and  is  testing  the 
systems  in  a  distributed  hardware  in  the  loop  environment  to  ensure  interoperability  across  the  AMN.  It  will  also  discuss  the 
applicability  to  other  theaters  that  may  need  to  implement  a  similar  process. 
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7:00  AM  -  6:30  PM 
7:00  AM  -  8:00  AM 
8:00  AM 

8:05  AM 


CONFERENCE  REGISTRATION  OPEN  -  2ND  LEVEL  REGISTRATION 
CONTINENTAL  BREAKFAST  IN  THE  DISPLAY  AREA  -  GRAND  SALONS  A-D 
OPENING  REMARKS  -  GRAND  SALONS  E-F 

►  Mr.  Sam  Campagna,  Assistant  Vice  President ,  Operations ,  NDIA 

TRIBUTE  TO  OUR  NATION  AND  WARFIGHTERS,  NATIONAL  ANTHEM 


SESSION  A:  CONFERENCE  WELCOME  &  KEYNOTES 

8:10  AM  WELCOME  AND  CONFERENCE  INTRODUCTORY  REMARKS 

►  Mr.  James  O’Bryon,  Chairman, ,  NDIA  T&E  Division;  The  O’Bryon  Group 

8:20  AM  CONFERENCE  KEYNOTE  ADDRESS 

►  Honorable  Dr.  J.  Michael  Gilmore,  Director ;  Operational  Test  &  Evaluation,  OSD 
Honorable  Dr.  J.  Michael  Gilmore  was  sworn  in  as  Director  of  Operational  Test  and  Evaluation  on  September 
23,  2009.  A  Presidential  appointee  confirmed  by  the  United  States  Senate,  he  serves  as  the  senior  advisor  to  the 
Secretary  of  Defense  on  operational  and  live  fire  test  and  evaluation  of  Department  of  Defense  weapon  systems. 

Prior  to  his  current  appointment,  he  was  the  Assistant  Director  for  National  Security  at  the  Congressional 
Budget  Office  (CBO),  and  was  responsible  for  CBO’s  National  Security  Division.  Dr.  Gilmore  is  a  former 
Deputy  Director  of  General  Purpose  Programs  within  the  Office  of  the  Secretary  of  Defense,  Program  Analysis 
and  Evaluation  (OSD(PA&E)).  Dr.  Gilmore  also  has  served  as  the  Division  Director  of  Operations  Analysis 
and  Procurement  Planning,  within  the  Office  of  the  Deputy  Director,  Resource  Analysis  and  as  an  Analyst  for 
Strategic  Defensive  and  Space  Programs  Division,  Office  of  the  Deputy  Director,  Strategic  and  Space  Programs. 

Dr.  Gilmore’s  service  with  Program  Analysis  and  Evaluation  covered  1 1  years.  Early  in  his  career,  Dr.  Gilmore 
worked  at  the  LLNL,  Livermore,  California  performing  research  in  their  magnetic  fusion  energy  program.  He 
has  also  worked  with  Falcon  Associates,  McLean,  VA,  and  the  McDonnell  Douglas  Washington  Studies  and 
Analysis  Group.  Dr.  Gilmore  is  a  graduate  of  MIT  where  he  earned  a  B.S.  in  Physics.  He  subsequently  earned  a 
M.S.  and  Ph.D.  in  Nuclear  Engineering  from  the  University  of  Wisconsin. 
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9:00  AM  GUEST  SPEAKER 


►  Honorable  Frank  Kendall,  Principal  Deputy  Under  Secretary  of  Defense,  AT&L,  OSD 


Mr.  Frank  Kendall  was  sworn  in  as  Principal  Deputy  Under  Secretary  of  Defense  for  Acquisition, 
Technology,  and  Logistics  (PDUSD(AT&L))  on  March  5,  2010.  In  his  role  as  PDUSD(AT&L),  Mr. 
Kendall  is  authorized  to  act  for  and  provide  assistance  to  the  Under  Secretary  of  Defense  for  Acquisition, 
Technology  &  Logistics  (USD  (AT&L)).  He  also  advises  and  assists  the  USD  (AT&L)  in  providing  staff 
advice  and  assistance  to  the  Secretary  of  Defense  on  the  acquisition  system;  research  and  development; 
modeling  and  simulation;  systems  engineering;  advanced  technology  and  developmental  test  and  evaluation. 
Within  government,  Mr.  Kendall  held  the  position  of  Director  of  Tactical  Warfare  Programs  in  the  Office  of 
the  Secretary  of  Defense  and  the  position  of  Assistant  Deputy  Under  Secretary  of  Defense  for  Strategic  Defense 
Systems.  Mr.  Kendall  was  also  Vice  President  of  Engineering  for  Raytheon  Company.  Mr.  Kendall  also  spent 
ten  years  on  active  duty  with  the  Army  serving  in  Germany,  teaching  Engineering  at  West  Point,  and  holding 
research  and  development  positions.  He  is  a  Distinguished  Graduate  of  the  U.S.  Military  Academy  at  West  Point 
and  he  holds  a  Masters  Degree  in  Aerospace  Engineering  from  California  Institute  of  Technology,  a  Master  of 
Business  Administration  degree  from  C.W.  Post  Center  of  Long  Island  University,  and  a  Juris  Doctoris  from 
Georgetown  University  Law  Center. 


9:30  AM 


HOMELAND  SECURITY  T&E  PERSPECTIVES 

►  Mr.  Gary  Carter,  Director,  Test  &  Evaluation  and  Standards  Division,  Department  of 
Homeland  Security 


10:00  AM  MORNING  BREAK  AND  NETWORKING  IN  THE  DISPLAY  AREA  -  GRAND  SALONS  A-D 


SESSION  B:  OTA’S  (OPERATIONAL  TEST  AGENCY’S)  ROUNDTABLE 

Session  B  Chair  and  Roundtable  Moderator:  Dr.  Catherine  Warner,  Science  Advisor,  DOT&E,  OSD 

10:30  AM  ROUNDTABLE 

►  MG  Genaro  Dellarocco,  USA,  Commander,  ATEC 

►  RADM  David  Dunaway,  USN,  Commander,  OPTEVFOR 

►  Maj  Gen  David  Eichhorn,  USAF,  Commander,  AFOTEC 

►  Col  David  Reeves,  USMC,  Commander,  MCOTEA 

►  COL  Joseph  Puett,  USA,  Commander,  JITC 

1 1 :30  AM  WALTER  W.  HOLLIS  HONORS  LUNCHEON:  PRESENTATION  FOR  OUTSTANDING  LIFETIME 

ACHIEVEMENT  IN  DEFENSE  TEST  &  EVALUATION  -  FLORIDA  SALONS  l-IV 

►  Dr.  James  N.  Walbert,  Chief  Scientist,  SURVICE  Engineering  Company 

Dr.  Walbert  has  more  than  35  years  of  DoD  T&E  and  related  experience  including  extensive  and  novel  work 
as  an  interior  and  exterior  ballistician,  a  vulnerability/lethality  tester  and  analyst,  a  materials  engineer,  and  an 
author  and  instructor.  From  1974  to  1978,  Dr.  Walbert  served  as  a  mathematician  and  test  director  for  the 
U.S.  Army  Material  Testing  Directorate,  where  he  planned,  analyzed,  evaluated,  and  assessed  a  wide  range 
of  engineering  test  programs.  From  1978  to  2000,  he  served  as  a  research  scientist/engineer  for  the  Ballistic 
Research  Laboratory  (and  then  the  Army  Research  Laboratory)  and  from  2001  to  2003,  Dr.  Walbert  served  as 
Chief  Scientist  for  the  DARPA  Future  Combat  Systems  Program  Office.  Since  joining  SURVICE  in  2003  as 
the  Chief  Scientist,  Dr.  Walbert  has  developed  numerous  analytical  processes  for  exploitation  of  ballistic  test 
data.  He  has  authored/co-authored  more  than  50  technical  publications  during  his  career,  including  the  AIAA- 
published  text  Fundamentals  of  Ground  Combat  System  Ballistic  Vulnerability/Lethality,  which  was  named  ARL’s 
Publication  of  the  Year  for  2009.  Based  on  this  text,  Dr.  Walbert  also  developed  and  teaches  a  highly  acclaimed 
basic  ballistic  vulnerability  course  to  Government  and  industry  practitioners  throughout  the  T&E  community. 

Dr.  Walbert  holds  a  B.S.,  M.S.,  and  Ph.D.  in  mathematics  all  from  the  University  of  Delaware. 
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1 1 :30  AM  LUNCHEON  GUEST  SPEAKER:  SOME  PROBLEMS  OF  CYBER  SECURITY 


►  Mr.  Robert  L.  Deitz,  former  General  Counsel ,  National  Security  Agency 
Robert  L.  Deitz  is  currently  Distinguished  Visiting  Professor  &  CIA  Officer-in-Residence  at  George  Mason 
University.  From  2006  until  February  2009  he  served  as  Senior  Councillor  to  the  Director  of  the  Central 
Intelligence  Agency.  From  September  1998  to  September  2006  he  was  the  General  Counsel  at  the  National 
Security  Agency  where  he  represented  the  NS  A  in  all  legal  matters.  Fie  has  also  held  positions  as  Acting  General 
Counsel  at  the  National  Geospatial-Intelligence  Agency  and  as  Acting  Deputy  General  Counsel,  Intelligence,  at 
the  Department  of  Defense.  Professor  Deitz  began  his  career  as  a  law  clerk  to  the  Fionorable  Justices  Douglas, 
Stewart,  and  White  of  the  United  States  Supreme  Court.  He  has  also  been  in  private  practice  and  was  Special 
Assistant  to  Deputy  Secretary  of  State  Warren  Christopher  and  to  Secretary  of  Health,  Education  and  Welfare 
Joseph  Califano  during  the  Carter  Administration.  Professor  Deitz  received  his  J.D.  (magna  cum  laude)  from 
Harvard  Law  School,  where  he  was  the  Supreme  Court  Note  and  Note  Editor  of  the  Harvard  Law  Review.  He 
received  an  M.P.A.  from  the  Woodrow  Wilson  School  of  Public  and  International  Affairs  at  Princeton  University, 
where  he  studied  international  politics  and  economics.  He  majored  in  English  literature  at  Middlebury  College 
where  he  received  a  B.A.  (cum  laude)  and  became  a  member  of  Phi  Beta  Kappa. 


SESSION  C:  ACQUISITION  REFORM  -  THE  IMPACT  ON  INDUSTRY 

Session  C  Chair:  Dr.  Suzanne  Beers,  MITRE  Corporation 

1 :1 5  PM  PENTAGON  RESPONSE  TO  CONGRESSIONAL  STRENGTHENING  OF  DT&E 

►  Mr.  Chris  DiPetto,  Principal  Deputy,  Developmental  Test  &  Evaluation 

1 :45  PM  REPORT  ON  NDIA’S  INDUSTRIAL  COMMITTEE  ON  TEST  &  EVALUATION  (IC0TE) 

►  Mr.  James  Ruma,  Chairman. ,  NDIA  ICOTE;  Vice  President ,  Engineering,  GDIS 
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5:30  PM  -  6:30  PM  RECEPTION  IN  THE  DISPLAY  AREA  -  GRAND  SALONS  A-D 
6:30  PM  CONFERENCE  ADJOURNED  FOR  THE  DAY 


WEDNESDAY,  MARCH  16, 2011 


7:00  AM  -  5:25  PM 
7:00  AM  -  8:00  AM 
8:00  AM 


CONFERENCE  REGISTRATION  OPEN  -  2ND  LEVEL  REGISTRATION 
CONTINENTAL  BREAKFAST  IN  THE  DISPLAY  AREA  -  GRAND  SALONS  A-D 
INTRODUCTION  AND  OPENING  REMARKS  -  GRAND  SALONS  E-F 

►  Mr.  Sam  Campagna,  Assistant  Vice  President ,  Operations ,  NDIA 
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WEDNESDAY,  MARCH  16, 2011  —  Continued 

SESSION  L:  A  RE-ENERGIZED  DT&E 

Session  L  Chair:  Mr.  John  Illgen,  Chairman, ,  NDIA  National  Board;  Northrop  Grumman 

8:05  AM  PANEL:  T&E:  SERVING  THE  WARFIGHTER  IN  A  COST-CONSTRAINED  ENVIRONMENT 

Panel  Moderator: 

►  Mr.  Chris  DiPetto,  Principal  Deputy,  Developmental  Test  &  Evaluation 
Panelists: 

►  Mr.  David  K.  Grimm,  Acting  Director,  Deputy  Under  Secretary  of  the  Army,  T&E  Office 

►  Mr.  Steve  Hutchison,  DISA  T&E  Executive 

►  Mr.  John  Manclark,  Air  Force  T&E  Executive 

►  Ms.  Amy  Markowich,  Navy  T&E  Executive 

►  Mr.  Tom  Wissink,  Director  of  Integration,  T&E,  Lockheed  Martin 

9:00  AM  SPECIAL  GUEST  PRESENTATION: 

EVALUATION  OF  THE  SINKING  OF  THE  CHEONAN  KOREAN  NAVAL  SHIP 

►  MG  Jong  Sung  Yoon,  Republic  of  Korea  Army  (Ret),  Leader  of  the  International  Investigation 
Team 

Rarely  does  one  have  the  opportunity  to  fully  investigate  the  circumstances  leading  up  to  the  attack  on  and 
sinking  of  a  warship  and  then  be  able  to  recover  the  ship  and  perform  an  extensive  international  investigation 
of  the  threat,  the  damage  and  casualties,  the  computer  modeling  of  the  damage  and  assessment  of  the  causes  and 
effects.  MG  Yoon  led  the  international  investigation  team  of  which  the  US  was  an  integral  part  into  the  sinking 
of  the  Republic  of  Korea’s  warship,  the  CHEONAN,  this  past  year.  His  insights  should  be  instructive  and  of  great 
interest  to  the  conference  attendees.  It  is  a  privilege  to  welcome  him  to  be  a  special  part  of  our  conference  this  year. 

In  addition,  MG  Yoon  will  be  joined  by  Dr.  Young  Shin,  Professor,  Naval  Postgraduate  School  and  visiting 
Professor,  Korean  Advanced  Institute  for  Science  and  Technology,  to  discuss  the  efforts  of  the  International 
Investigation  Team  addressing  the  CHEONAN  sinking. 

MG  Jong-Sung  Yoon  was  born  on  April  4th,  1975  in  Inje-gun,  Gangwon-do,  Korea.  In  1981,  he  received  his  B.S. 
from  the  Korea  Military  Academy  (37th);  in  1999,  MG  Yoon  received  his  M.S.  in  Science  of  public  administration 
from  Dongguk  University;  in  2008,  he  received  his  Ph.D.  in  Politics  from  Myongji  University. 

10:00  AM  MORNING  BREAK  AND  NETWORKING  IN  THE  DISPLAY  AREA  -  GRAND  SALONS  A-D 

SESSION  M:  RESPONSIVE  AND  AGILE  INFORMATION  SYSTEMS  T&E  PANEL 

Session  M  Chair  and  Panel  Moderator:  Dr.  Steve  Kimmel,  Chairman,  NDIA  C4ISR  Division;  Senior  Vice  President, 

Alion  Science  &  Technology 

10:30  AM  PANEL 

Panelists: 

►  Dr.  Steven  Hutchison,  Director  T&E,  DISA 

►  Dr.  James  Streilein,  Deputy  Director,  Net-Centric  and  Space  Systems,  DOT&E 

►  Ms.  Darleen  Mosser-Kerner,  Deputy  Director,  Capabilities  Development,  Office  of  the  Director, 
DT&E 

►  Mr.  Eustace  King,  Chief,  Acquisition  and  Technology,  DOD-CIO/NII 
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WEDNESDAY,  MARCH  16, 2011 

1 1 :30  AM  LUNCHEON  -  TESTER  OF  THE  YEAR  AWARDS  -  FLORIDA  SALONS  l-IV 

This  awards  event  is  a  highlight  of  our  annual  conference  since  it  provides  the  opportunity  to 
recognize  outstanding  achievement  in  test  and  evaluation  by  members  of  our  armed  forces,  DoD 
civilians  and  DoD  contractors.  Furthermore,  what  makes  these  awards  particularly  noteworthy  is 
that  the  selections  are  made  by  the  organizations  of  those  being  recognized.  Congratulations  to  all 
who  are  being  recognized  for  their  2010  accomplishments. 

SESSION  N:  IMPROVING  THE  T&E  PROCESS 

Session  N  Chair:  Dr.  Lowell  Tonnessen,  IDA 

1 :1 5  PM  T&E  AND  MISSION  ASSURANCE 

►  Mr.  James  W.  Wade,  Vice  President ,  Raytheon  Company 

1 :45  PM  S0C0M  T&E  PERSPECTIVES:  SERVING  THE  WARFIGHTER 

►  LTC  Kevin  Vanyo,  USA,  USSOCOM J8-0 

►  Mr.  Robert  D.  Werner,  Jr.,  Senior  Test  Officer ;  USSOCOM J8-0 

2:1 5  PM  -  5:25  PM  CONCURRENT  SESSIONS  0  -  V 
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SESSION  O 

T&E  M&S  for  Specific 
Applications 

Grand  Salon  G 

RADM  Bert  Johnston, 
USN  (Ret),  Wyle 
Corporation 

1 1560  -  A  Comprehensive 

Approach  to  Characterizing 
the  Hazards  of  Explosive 
Countermeasures  With  Respect  to 
Dismounted  Troops 

Mr.  Stephen  Swann,  U.S.  Army 
Research  Laboratory 

1 1 529  -  Expanding  Use  of  the 
Probability  of  Raid  Annihilation 
(PRA)  Test  Bed  Across  the  Ship 
Self-Defense  Enterprise 

Mr.  Richard  Lawrence,  AVW 
Technologies 

SESSION  P 
Approaches  to 
Organizing  an 
Effective  M&S 
Program  in  Support 
of  T&E 

Grand  Salon  H 

Mr.  Britt  Bray,  DRS 
Corporation 

1 1500  -  Modeling  and  Simulation 
for  Mission-Based  Test  and 
Evaluation  (MBT&E) 

Mrs.  Beth  Ward,  U.S.  Army 

Research  Laboratory 

1 1476  -  A  Paradigm  for  Modeling  and  Simulation  in  support  of  Mission- 
Based  Test  and  Evaluation 

Dr.  James  Walbert,  SURVICE  Engineering  Company 

SESSION  Q 
T&E 

Instrumentation 

Infrastructure 

-  Maximum 

Utilization  of 
Available  Resources 

Grand  Salon  I 

Mr.  Dick  Dickson, 
Tybrin  Corporation 

1 1497  -  Joint  Mission 

Environment  Test  Capability 
(JMETC):  Improving  Distributed 
Capabilities 

Mr.  Chip  Ferguson,  JMETC 

11508  -  U.S.N.  RDTE  Project 
Support  Aircraft 

Mr.  Charles  Myers,  U.S.  Navy, 
NAWCAD 

1 1 626  -  Dugway  Proving  Ground 
as  the  MRTFB  Chem  Bio  Activity 

Ms.  Jean  Baker,  U.S.  Army  Dugway 
Proving  Ground 

SESSION  R 
Applications 
of  Design  of 
Experiments  (DoE) 
to  T&E 

Grand  Salon  J 

Mr.  Martin  Woznica, 
Raytheon  Company 

1 1677  -  Using  Design  of 
Experiments  (DoE)  to  Integrate 
Developmental  and  Operational 
T&E 

Dr.  Mark  Kiemele,  Air  Academy 
Associates 

1 1 549  -  Probability  Driven 
Experiments  Design  for 

Autonomous  Systems 

Mr.  Troy  Jones,  Charles  Stark 

Draper  Laboratory 

11532  -  Design  of  Experiments: 
Managing  Expectations 

Mr.  James  Carpenter,  AVW 
Technologies,  Lnc. 
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5:25  PM  CONFERENCE  ADJOURNED  FOR  THE  DAY 

THURSDAY,  MARCH  17, 2011 

7:00  AM  - 12:00  NOON  CONFERENCE  REGISTRATION  OPEN  -  2ND  LEVEL  REGISTRATION 
7:00  AM  -  8:00  AM  CONTINENTAL  BREAKFAST  IN  THE  DISPLAY  AREA  -  GRAND  SALONS  A-D 
8:00  AM  INTRODUCTION  AND  OPENING  REMARKS  -  GRAND  SALONS  E-F 

►  Mr.  Sam  Campagna,  Assistant  Vice  President ,  Operations ,  NDIA 
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THURSDAY,  MARCH  1 7,  201 1  —  Continued 

SESSION  W:  TEST  DESIGN,  TEST  CURRICULA  AND  STANDARDS 

Session  W  Chair:  Dr.  Paul  Deitz,  former  Technical  Director,  AMSAA 

8:05  AM  SYSTEMS  ENGINEERING  PLANS:  HOW  TO  RECOGNIZE  PROBLEMS,  SET  GOALS  AND 


8:30  AM 

IMPLEMENT  IMPROVEMENTS 

►  Dr.  Don  McKeon,  Defense  Acquisition  University 

11690  -  DOING  MORE  WITHOUT  MORE  -  SCIENTIFIC  T&E  DESIGN  METHODOLOGIES 
(STED  IN  DoD  WEAPONS  SYSTEMS  AQUISITI0N) 

►  Ms.  Darken  Mosser-Kerner,  Deputy  Director,  Capabilities  Development,  Office  of  the  Director, 
DT&E 

8:55  AM 

WHAT  ARE  WE  TEACHING  OUR  PMs  AND  ACQUISITION  PROFESSIONALS  ABOUT  T&E? 

►  Col  Michael  Bohn,  USMC  (Ret),  Faculty,  Defense  Acquisition  University 

9:10  AM 

REPORT  ON  STANDARDS  FOR  DT&E 

►  CDR  Ernest  Swauger,  USN  (Ret ) ,  JPEO-CBD/Chiefi  CM/HD  Systems  IPAT 

9:35  AM 

11663  -  EFFECTIVE  COMBAT  DATA  COLLECTION  &  APPLICABILITY  TO  T&E 

►  LtCol  Michael  Kennedy,  USMC,  Expeditionary  Test  Division,  MCOTEA 

10:00  AM 

MORNING  BREAK  AND  NETWORKING  IN  THE  DISPLAY  AREA  -  GRAND  SALONS  A-D 

10:30  AM 

-  2:00  PM  BREAKDOWN  OF  DISPLAYS 

SESSION  X:  CONFERENCE  SYNOPSIS  FORUM 

Session  X  Chair:  Dr.  Paul  Deitz ,  former  Technical  Director,  AMSAA 

1 0:30  AM  1 1 651  -  TEST  &  EVALUATION  ISSUES  FOR  SYSTEMS  OF  SYSTEMS  (SoS):  CREATING 


10:55  AM 

SLEEP  AIDS  FOR  THOSE  SLEEPLESS  NIGHTS 

►  Dr.  Beth  Wilson,  Principal  Engineering  Fellow,  Raytheon  Company 

11569  -  T&E  -  GUARDING  THE  REQUIREMENTS  INTENT 

►  Mr.  Steve  Scukanec,  Senior  Test  Engineer,  Northrop  Grumman  Aerospace  Sector 

11:25  AM 

CONFERENCE  SYNTHESIS  PANEL 

►  Dr.  Suzanne  Beers,  T&E  Group  Leader,  MITRE  Corporation 

►  Mr.  Britt  Bray,  Military  Analyst  and  Department  Manager,  DRC  Corporation 

►  Mr.  Brian  Simmons,  Executive  Technical  Director/ Deputy  to  the  Commander,  U.S.  Army  Test 
and  Evaluation  Command 

►  Dr.  James  Streilein,  Deputy  Director,  OSD,  DOT&E 

►  Dr.  Catherine  Warner,  Science  Advisor,  OSD,  DOT&E 

11:55  AM 

CLOSING  REMARKS 

►  Mr.  James  O’Bryon,  Chairman,  NDIA  T&E  Division;  The  O’Bryon  Group 

1 2:00  NOON  CONFERENCE  ADJOURNS 
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Who  We  Are 
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Raytheon  Corporation 
Raytheon  Missile  Systems  (RMS) 

Engineering 

Systems  Test  Directorate 

■  System  Test  Director  reports  to  RMS  VP  of  Engineering 

■  Three  Major  Elements  of  Systems  Test 

-  Hardware  in  the  Loop  (HWIL)  simulators 

-  Special  Test  Equipment 

-  Integration  and  Verification 
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Command  Media 
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■  Numerous  sources  for 
command  media 

-  Corporate:  Integrated  Product 
Development  System 

-  Business  Unit  (RMS)  Directives 

-  Organizational  Directives 

-  Program  Directives 


IPDS 

Integrated  Product  Development  Process 


Process  Assets  Library 


How  We  Do  Our  Job 
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One  Page  Process  (1PP)  tailored  to  discipline 


Integration  &  Verification 
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Hardware  in  the  Loop 


Designed  to  follow  the  life  cycle  of  a  program  from 

proposal  to  production 
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One  Page  Process  Web  Page 


Raytheon 


■  Acts  as  a  front  end  that  provides 
linkages  back  to  command  media 

■  Includes  Test  Equipment,  HWIL, 
l&V 

-  continue  to  work  to  incorporate  all  functions 
in  the  directorate 


Communication 


IVC  Design  Guide 


IVC  Processes 
and  Tools 


IVC  One  Page 
Process 


Integration  and 
Test  Fundamentals 


5 Why  Template 


3/17/2011  5 


Raytheon 


Details  on  l&V  Process 


System 

Requirements 


FVoposat/SOW 


RFP&SOW 

Fumjjng 

Document 

TEMP 


ICD's 


Preliminary  Specs 


System 

Requirements 


V&VMatw 


Design  Specs 


Acronyms 


TM  System 
Requirements 


Hardware 

Resources 
fTE,  GFE.  Facilities) 

DMAP 

□  ExternaMnputs 

□  IPOS  Stages 

□  Major  Process  Steps 

□  Reviews 

□  Gate  Reviews 

□  External  Outputs 


Feedback 


1.0  Business 
Strategy 


BdjProposal 

Development 


2  0  Management  &  Control 


Project  Management 


3.0  Requirements 
&  Architecture 


Requirements 


&V  Capture 
Management 
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ITBP 


4.0  Design  &  Development 

Preliminary 

Design 
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^ j 

[  System  Integraton  j 

l  Flo*  j 

s 

TestRans 

s  J 

S  V 

Data  Analysis 
Ran 

\  / 
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HUIWAUM 

Ship  &Test 

Re  ad  mess  Peer 

Review 


Test  Dry  Runs 


BOE«OM 


White  Sheel 

Review 


Preliminary  Desgn  Detailed  Design 

Peer  Review(s)  Peer  Reviews) 


CTS/TRR 


Gate  4 

Proposal  Review 


Gate  5  Start-up 
Review 


Gate  6 

jSystem  Requirements 
Review 


Gate  7 
FVeliminary  Design 
Review 


Gate8  ' 
Critical  Design 
^  Review  ^ 


Post  Test 

Rewew(s) 


Gala 0 

Test  Readiness 


i&V  Strategy, 
IMS. 

BOE/RQM, 
Formal  Proposal 
inputs 
HUM.'AUM 


ITEP  Draft 
Updated  l&V  IMS. 
Org.  Chart. 
Manpower  Plan 


*■  s 

5.0  Integration,  Verification  &  Validation 

Integration 

Venfication 

Validation 

V 

_ / 

/  ^ 

Integration 

Tests 

\  / 

/ 

Test 

Evenl 

k  / 

AUR 

Integration 
\  j 

{  \ 

Test 

FTooedures 

S  / 

i  »  a 

Data 

Analysis 

X  / 

f  \ 

Ratform 

Integration 

V  V 

UUT  Pedigree 

Documentation 
v.  J 

Test 

Reports 

X _ 

/  ' 

Field/Flight 

Test 

v  / 

7.0  Support 


OT&E 


OT  Build 
&Test 


OT&E  Support 


Updated  ITER 
Gate  6 1 SRR 
DataPkg 

System 

integration  Flow. 
Gate  7  /  PDR 

Data  Fkg 

Updated 
HUM/AUM, 
TestRan(s) , 

Data  dialysis 

Ran  GateS/  CDR 
Data  Pkg 

Test  Procedures), 
CTS/TRR 

Data  Pkgfs) 

Venfication  Test 
Reports, 

Gate 9  DataPkg 

Validation 

Test  Reports 

’■ - 
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Input  Across  Top 


Raytheon 


System 

Requirements 

FVoposal/$0W 

TEMP 

RFP&SGW 

Funding 

Document 

ICD’s 

Preliminary  Specs 


System 

Requirements 


V&V  Mlafw 


Design  Specs 


TM  System 
Requ»rements 


Hardware 

Resources 
(IE,  GFE.  Facilities) 

DMAP 

Inputs:  Documentation/Products  which  are  expected  to  be  received  from  outside 
the  l&V  organization 


3/17/2011 


Stages 


Raytheon 


1.0  Business 
Strategy 


BdJProposal 

Development 


2  0  Management  &  Control 


Project  Management 


3.0  Requirements 
& Architecture . 


Requirements 


4.0  Design  &  Development 


Preliminary 

Design 


Detailed 

Design 


5.0  Integration,  Verification  &  Validator 


integration 


Verification 


Validation 


The  One  Page  Process  is  structured  around  the  existing  IPDS  process 


liYTEtiKATED  PHOOITT  DEVELOPMENT  PlMJl’EHH 


'Stage  1:  Business 
Planning  Strategy  1 
Execution 

L-~  i joq>oij02jd3>04 

"  Stage  2:  Program  Leadership,  Management 
and  Control 

i‘Q5j  (lljf 

Stage  3: 

Requirements  & 
Architecture 
Development 

@L 

The  stages  of  Uir 
Integrated  Produtd 
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CIica  rr  3  numbered  Gate 
icon  (®)  to  view  its  task 
de&cfiplioiT 


Stage  4:  Design 
and  Development 

l  lg>  08 


i 


Stage  5: 

r  Stage  6: 

Integration. 

Production 

Verification  & 

and 

Validation 

Deployment 

Stage  7: 
Operations  and  Support 
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Activities 


Raytheon 


Activities:  The  core  tasks  that  l&V  accomplishes  during  the  course  of  a  program. 
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Note  that  activities  can  be  repeated  across  multiple  IPDS  stages  but  are 
generally  listed  in  the  stage  in  which  they  are  first  completed 
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Activity  Sheets 


3  External  Inputs 
3  IPOS  Stages 
3  Major  Process  Steps 


Feedback 


5.0  Integration,  Verification  &  Validation 

|  Integration  | 

|  Venfcabon  | 

|  Validaion  | 

f  “sr  ) 

f  Test 

[  Event 

f  AUR 

[  Integration 

[  Test  | 

[  Procedures  j 

(  Data 

I  Analyas 

f  Platform 

1  Integrabon 

\  UUT  Pedigree  ] 
{  Documentafion  j 

(  Test 

[  Reports 

[  FieldIFlight 

I  0T&E  I 


Describe  work  activities  in  detail 

Links  are  provided  to  command 
media,  templates,  examples, 
etc. 


«ier  Review(s)  Peer  Kevieefs)  Woi 


ITEP  Draft, 
pdated  l&VIMS. 
Org.  Chart. 


Updated  US’ 
Gate  6 1 SRR 
DataPkg 


tetegrabon  Flow, 
Gale  f  I POR 
Datartg 


CiMintnl  Rflqulrnfnnnnc: 


eu  kpi  udurl  Dusli  ipliun: 


llel  DoLiJiTMrt^ 


Isr  nr  tnsl  npsracnr  dpdarkcl  incIryrtirHia  on  h(w  eg  purlonn  she 
4*  hdirt  lire  ussmtiHldtl  Inal  JHiKib  The  apprtnnl  ul  brtl  proLteilum 
Ci  (ha  pKtfWf  ffi  CMrt*  Ml  HST-gT-IOJ. 


Id  D'lJ  UetH  l>tti  Will 


,  i  nummary  <5t  Ihu  fqiKCmls  Of  Igsl  prcKvduiya  -i  bultnr 

i.  ftmooucnoM 

1 1  ii«o«al 
* 1  RifettKtt 
USnpt 
f  t-nri  Flaw 

t  j.  Tijol  RudiMintn  brblm 

2  CONfjatufiAI'lOH 

2 1  Unit  Unhr  f*H  (jUU  f  j  C  tin  mm  radon 

1 2  Tnsl  Equamiunl  Cuntiyu' jUron 


nr  (w  Irtl  praiKiluiea  '»  CnnlvtKd  n  RfST-ST-lKA  for  v,  j-|^.  m 


3  JUlTTfir  E^Vlty  G,w«nS 

social  rest  iNsmocTiOHS 

4  1  Security 

i  ?TlWS  RurOna 
4  :r  Pnge  adorn  Chuigaa 
4 ,4  O-ijprd  Purcrtdurii  ttareHCpmenl 
■S  S  i  «[  lit  Psulinipanl-i 
4  1 1«H  6llEtl*fS 

Id  T«a  Pvaporatrsnc 


IRihi  K 


PiOwduuia  aJibuld  b* 
HST^V103. 


Dkimum  Nunetaer: 


Pilitsiie  Systems 


Instructions 


T'iu«t:  14  v  s«r  Pronc-durf  Orfoi 


tn*  IBtewiig  tei  it  itrmmrrwnjJrid  diitent?  rr*  *  gofer  tt  prrtrfedura  ArtdllKmJ  pnJpHlMi*  Otl>At»  wet  pnetakfed  in 

Itie  appender  a«  (Pus  document 


0.0  COVER  SHEET 

lnslruflioislMSt^t-103  ra*eia  life  .kHjiI  nU  ret  17m  enitf  1I3.  tw  letxl  peeira'dilrei 


this  is  a  drier  eversne*  or  Bte  program  and  purpose  or  Bits  document 
1  .£  References 


A  ksfcng  of  all  references  tMt  arc  used  w*Nn  the  procedure  la  included.  Typical  references  may 
hwiidtt  f*OLinfe  jrtquirftiiE-iih;  (plans}.  rmiHgiidiran  da*£o$lji*  fetH  equprriefri!  liafcd.  (Ml  reiyuu  rnierib 
dptinnerfla.  rie  The  refttrtjrMyt  IqflaigjAd  Irniufe  rjoi.airoerrl  ■niiiiUttf  a  Iflltt.  uri-iKtn  did*  ferCtfity 

classification  and  a  d  esc  nab  on  of  Ihe  data  content  4nd  me  iTI  is  not  obvious  tfom  ine  blfc 


Scope  Hue  test  procedure  contains  a  scope  or  purpose  mat  dellnes  the  product  cwUlguraUons  and 
tOMtfc  mutied  By  lit*  pTOCMtfee  It  Any  ■Otlfie  wm  mum  not  Wen  prwrtw.  \rn  njmHtart  hi  flteJtelf 


3/17/2011  10 


Reviews 


Raytheon 


Reviews  are  conducted  before  and  after  major  activities  and  stages 

Note  that  some  reviews  such  as  CTS/TRR  can  be  repeated  multiple  times 
during  the  course  of  the  program 
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: System  Requirement^  FVelimin ary  Design  I  Critical  Design 
Review  Review  Review 


Gate  9 

Test  Readiness 


3/17/2011 


Outputs 


Raytheon 


The  products  which  are  created  during  the  course  of  the  given  stage 


o 


l&V  Strategy, 

ITEP  Draft 

IMS.  ' 

Updated  l&V  IMS. 

BOE/ROM, 

Org.  Chart. 

Format  Proposal 

Manpower  Plan 

Inputs 

HUM/AUM 

Updated  IfEP 
Gate  6/SRR 
Data  Pkg 


System 

Updated 

TestProc8dure(s), 

Verification  Test 

Integration  Flow 

HUM/AUM. 

CTS/TRR 

Reports, 

Gate  7/PDR 

Test  Ran (s), 

Data  Ptatfs) 

Gate 9  DataFVg 

DalaFVg 

Data  Analysis 

Ran  Gate  8 1  CDR 
Data  Pkg 

Valtd  a  bon 
Test  Reports 


3/17/2011 


Conclusion 


Raytheon 


■  All  linked  material  is  controlled  and  approved  by 
management  before  release 

■  Future  is  to  expand  it  to  expand  its  use  in  the  directorate  and 
to  other  areas  of  engineering 


3/17/2011 
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What  is  Mission-Based  Modeling  and  Simulation? 

The  Value  of  Intermediate  Results 

Applicability,  Precision,  and  Accuracy 

So,  exactly  what’s  in  that  Field  and  Test  Data? 

(and  therefore,  what  should  be  in  the  Simulation  Output?) 

What  constitutes  a  “good”  model? 

If  you  don’t  have  a  road  map,  don’t  take  the  M&S  trip 
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What  is  Mission-Based 
Modeling  and  Simulation? 
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All  T&E  Is  (should  be)  Mission-Based 
All  M&S  is  (should  be)  Mission-Based 


The  following  three  (evaluation)  missions  require 
three  different  levels  of  data  to  evaluate,  and 
three  different  levels  of  modeling  to  simulate: 

See  if  threat  x  can  perforate  target  y 

See  by  how  much  threat  x  perforates  target  y 

See  how  threat  x  perforates  target  y 


If  I  complete  a  certain  (evaluation)  mission ,  haven’t  I 
completed  each  (evaluation)  mission  above  it? 

NOT  NECESSARILY!! 
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Perforation 

(Y/N) 

Residual  Mass, 
Velocity,  etc. 

Retailing, 
Plugging,  etc. 

Striking  Velocity, 
Obliquity 


Mission-Based  Test  &  Evaluation 


6.  Context,  Environment  (Military,  Civil,  Physical,  etc.) 


7.  OWNFOR  Why  =  Purpose,  Mission 


7.  OPFOR  Why  =  Purpose,  Mission 


f  I  4.  Tasks,  Operations!  \ 

o4 

Capabilities  °WNFOR 


t 


4.  Tasks,  Operations  ■ 


°2,3  2.  Components, 


WQ 

B  OPFOR  I  Functions, 

|  Capabilities 

2.  Components,  ®2,3 
Forces 


Enough? 


By  How 
Much? 


How? 


There  are  many  possible  paths 


See  if  threat  x  can 
perforate  target  y 


See  by  how  much  threat 
x  perforates  target  y 


Mission, 

Task 


Remaining 

Capability 


Component 

Damage 


See  how  threat  x 
perforates  target  y 
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The  Value  of 
Intermediate  Results 
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Mission-Based  Test  &  Evaluation 


THREAT 

VELOCITY 

MATERIAL 

MATERIAL 

V50 

TYPE 

(Ft/Sec) 

TYPE 

THICKNESS  (Inches) 

(Ft/Sec) 

X 

500 

ALUMINUM  5083 

0.1250 

288.39 

THREAT 

VELOCITY 

MATERIAL 

MATERIAL 

V50 

RESIDUAL 

RESIDUAL 

YAW 

TYPE 

(Ft/Sec) 

TYPE 

THICKNESS  (Inches) 

(Ft/Sec) 

VELOCITY  (Ft/Sec) 

MASS  (Grains) 

(Degrees) 

X 

500 

ALUMINUM  5083 

0.1250 

288.39 

408.45 

745.00 

0.49 

PLUGGING 
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Less  “Precise” 
More  General 


Analysis 


More  “Precise” 
Less  General 


Less  Complex 
Less  Costly 


Testing 


THREAT  I  VELOCITYl MATERIAL  I  MATERIAL  I  V50 

TYPE  |  (Ft/Sec)  |  TYPE  |  THICKNESS  (Inches)  |  (Ft/Sec) 


500 


ALUMINUM  5083 


0.1250 


288.39 


THREAT 

1  VELOCITYl 

1  MATERIAL  1 

MATERIAL  1  V50  1 

RESIDUAL  I 

RESIDUAL  1 

1  YAW 

TYPE 

|  (Ft/Sec)  | 

TYPE  | 

|  THICKNESS  (1  nches)  |  (Ft/Sec) | 

VELOCITY  (Ft/Sec)  | 

MASS  (Grains)  | 

|  (Degrees) 

X 

500 

ALUMINUM  5083 

0.1250  288.39 

408.45 

745.00 

0.49 

-# 

BRITTLE  FRACTURE  RADIAL  FRACTURE  DUCTILE  HOLE  GROWTH 

H5*H§ 


FRAGMENTATION 


More  Complex 
More  Costly 


£ 
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The  dangers  of  a  very  specific  model 


More  Universally  Accurate 

A  President  was  Elected 

\  (very  general,  but  correct) 


M&S 


Applicability 


Less  Universally  Accurate 


Thomas  Dewey  was 
Elected  President 

(very  specific,  but  incorrect) 
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The  dangers  of  a  very  specific  model 


This  very  precise  model 
does  not  explain  how  the 
President  was  elected. 

The  model  of  at  least  one 
of  the  mappings  is  flawed; 
everything  that  follows  is 
probably  incorrect,  such  as  by 
how  much  (how  many  votes). 

If  the  prediction  was  precisely  incorrect  because  17 
precincts  voted  the  opposite  from  the  assumption, 
then  “tweaking”  the  model  to  change  the  way  those 
17  precincts  vote  may  or  may  not  to  produce  “better” 
results  in  the  next  election. 


More  Universally  Accurate 


M&S 


Applicability 


A  President  was  Elected 


Less  Universally  Accurate 


Thomas  Dewey  was 
Elected  President 
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Applicability,  Precision 
and  Accuracy 
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MY 


A 


Walbert’s  view  of  the  world** 


**Not  to  scale 
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MY 


A 


Walbert’s  view  of  the  world 


Perception 


Calculation 


Measurement 
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Possible  Disparity 
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Walbert’s  view  of  the  world 


The  greater  the  precision 
in  any  one  of  the  domains,  § 
the  more  likely  it  is  that  it  | 
will  disagree  with  the  other  | 
domains.  £ 


0 


Increasing  Precision 
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Walbert’s  view  of  the  world: 

An  Event  (Field,  Test,  Simulation) 


If  the  data  points  from  the  domains  are  at  differing  levels  of 
precision  (granularity),  then  comparison  is  “difficult.” 


The  location  of  the  data  point  on  the  “Truth”  curve  is  unknown 
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So,  exactly  what’s  in  that 
Field  and  Test  Data? 

(and  therefore,  what  should  be 
in  the  Simulation  Output?) 
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An  Example:  Craters 


Truth 


Perception/ 
Field  Data 


Measurement/ 
Test  Data 


Calculation 


SOIL 

TYPE 

MOISTURE 
CONTENT 
(%  of  Satur) 

ENERGETIC 

MATERIAL 

CHARGE 

WEIGHT 

lbs 

DEPTH  of 
BURST 
feet 

MEASURED 

DIAMETER 

feet 

MEASURED 

DEPTH 

feet 

Mixed  Soil 

100 

XXX 

21.00 

1.75 

12.80 

3.95 

SOIL 

TYPE 

MOISTURE 
CONTENT 
(%  of  Satur) 

ENERGETIC 

MATERIAL 

CHARGE 

WEIGHT 

lbs 

DEPTH  of 
BURST 
feet 

ESTIMATED 

DIAMETER 

feet 

ESTIMATED 

DEPTH 

feet 

Mixed  Soil 

100 

XXX 

20.28 

2.06 

13.00 

5.07 

22.49 

8.18 

13.00 

4.23 

44.09 

0.00 

11.74 

4.00 

64.82 

13.16 

16.09 

4.00 

MEASURED 

DIAMETER 

feet 

MEASURED 

DEPTH 

feet 

13.00 

4.00 
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Match  the  analysis  to  the  data  content 


00 

+  ^  { ancos(nt)  +  bnsin(nt) } 

n=l 


Fourier  Series  Coefficients 
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MY 


A 


Match  the  analysis  to  the  data  content 
An  example:  Acceleration  Data 
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MY 


A 


Match  the  analysis  to  the  data  content 
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MY 
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Match  the  analysis  to  the  data  content 
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You  can’t  get 
blood  from  a 
turnip! 
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What  constitutes  a 
“good”  model? 
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HOW  TO  TELL  A  “GOOD”  MODEL 
FROM  A  “BAD”  MODEL 


Which  question  is  more  appropriate? 

1 )  How  well  did  the  model  predict  the  outcome  of  the  test? 

2)  Was  the  outcome  of  the  test  a  member  of  the  population 
of  possible  outcomes  predicted  by  the  model? 


If  my  model  gets  the  “right”  answer,  doesn’t  that  mean 
I  understand  the  phenomenon? 

NOT  NECESSARILY!! 
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MY 


A 


HOW  TO  TELL  A  “GOOD”  MODEL 
FROM  A  “BAD”  MODEL 
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MY 


A 


HOW  TO  TELL  A  “GOOD”  MODEL 
FROM  A  “BAD”  MODEL 


Average  Solar  Radiation  (Calories  per  sq  cm  per  minute) 
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A 


SO,  YOU  STILL  THINK  THAT’S  FUNNY? 


Percent  +/-  Normal 

n  o  cn  o  cn  o  cn 

Change  in  Sunspot  Area/30  and  Index  of  Total  US  Production  (Excluding  Food)  by  Year 

I  \  / 

■ ■ i_ i 

■ ■ i_ i_ i 

■ ■ i_ i 

■ ■ i_ i_ i 

■ ■ i_ i 

■ ■ i_ i_ i 

■ ■ i_ i 

■ ■ i_ i 

- 1  u 

1875  1880  1885 

1890  1895  1900  1905  1910  1915  1920  1925  1930 

Year 

Correlation  coefficient  =  0.76 

t-value  =  -1.15  (not  in  critical  region,  no  stat.  sign.  diff.  at  5%  level) 


Source:  Center  for  Cosmic  and  Terrestrial  Research,  MIT,  1937 
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There  may  be  no  path  at  all! 


Does  “Correlation”  mean  the  same 
thing  as  “Cause  and  Effect?” 
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If  you  don’t  have  a  road  map 
don’t  take  the  M&S  trip 
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An  Un-verifiable  Model 


Future  Combat  Systems 
Network  Conceptual  Representation 


The  mission  is  to  see  if  the  network  does  its  job 

(i.e.:  is  effective) 


An  Un-verifiable  Model 


The  platforms  and  sensors  are  the  “components”  at  Level  2 


The  applications  represent  the  tasks  or  operations  at  Level  4 


The  services  represent  the  capabilities  at  Level  3 


The  standards  represent  the  context  at  Level  6 


^ranspoi^ 


Standard^ 


This  conceptual  representation  has  no  reasonable  logic  flow 
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MY 


A 


An  Un-verifiable  Model,  Made  Verifiable 


If  instead,  we  use  the  following: 

Node  Functional  Logic  Flow 


This  encompasses  all  requisite  network  functions... 
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r  An  Un-verifiable  Model,  Made  Verifiable 


and  follows  a  logical  progression: 


The  utilize  layer  corresponds  to  task  execution  at  Level  4 


Utilize 


The  synthesize  layer  corresponds  to  the  capabilities  at  Level  3 


The  process  layer  corresponds  to  the  “components”  at  Level  2 


rocess 


If  it  all  worked  satisfactorily  each  time,  the  mission  was 
completed.  If  not,  the  mission  wasn’t  completed. 
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r  An  Un-verifiable  Model,  Made  Verifiable 


If  it  didn’t  work,  why  not? 


Did  the  node 

1 )  Get  the  information  it  needed  when  it  needed  it? 

2)  Understand  the  information? 

3)  Process  the  information  successfully? 

4)  Use  the  information? 


All  of  these  questions  assume  the  information  was  in  the 
appropriate  context. 
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MY 


A 


An  Un-verifiable  Model,  Made  Verifiable 


If  it  didn’t  work,  why  not? 

Did  the  node 


The  utilize  layer  corresponds  to  task  execution  at  Level  4 


The  synthesize  layer  corresponds  to  the  capabilities  at  Level  3 


Utilize 


The  process  layer  corresponds  to  the  “components”  at  Level  2 


Process 


1) 

2) 
3) 


Get  the  information  it  needed  when  it  needed  it? 


Understandthe  information? _ 

Did  the  Platforms  and  Sensors  Layer  work  Properly? 

Proces^h^nfomiatior^uccessfullv? 

Did  the  Services  Layer  work  Properly? 


4)  Use  th^nfonriation^^^^^^^^^^^ 

Did  the  Applications  Layer  work  Properly? 


All  of  these  questions  assume  the  information  was  in  the 


appropriate  context.  Did  the  Standards  Layer  work  Properly? 
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The  Missions  and  Means  Framework 


6.  Context,  Environment  (Military,  Civil,  Physical,  etc.) 


7.  OWNFOR  Why  =  Purpose,  Mission 


7.  OPFOR  Why  =  Purpose,  Mission 


The  blue  arrows  indicate  “Planning” 
The  red  arrows  indicate  “Execution” 
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MY 
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A  Very  Old  Concept 


Mission 


Analytical 

Plan 


Data 

Requirements 


Test  Plan 


The  analytical  plan  is  based  on  the  mission. 

The  data  requirements  are  based  on  the  analytical  plan. 
The  test  plan  is  based  on  the  data  requirements. 

ANY  OTHER  ORDER  FOR  THESE  EVENTS  IS  NONSENSE! 
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The  Paradigm 


Organize  the  M&S  and  T&E  using  the  logic  flow  of  MMF. 
Determine  the  number  of  levels  (intermediate  outputs)  required. 
Align  the  data  collection  (instrumentation)  with  the  levels. 

Develop  the  M&S  to  output  the  same  intermediate  levels  (values). 


tGSt 

Don’t  i  .  ,  f  more  detail  than  you  need,  and 
model 


don’t 


model 

test 


more  detail  than  you 


test 
model 


39 


Points  to  Ponder 


Should  we  always  design  a 


*es*  -that  fits  all  missions? 

model 


(just  in  case... Scope,  Time,  Budget) 

Is  it  better  to  be  precisely  incorrect  or  approximately  correct ? 

(“Dewey  Beats  Truman”  vs  “A  President  was  Elected”) 

(If  the  test  data  value  is  1.2  and  the  simulation  output  is  1.23564, 

which  value  is  more  nearly  correct?) 

Are  we  doing  a  certain  level  of  M&S  because  we  can,  or  because 
we  need  it  to  answer  the  “mission  accomplished”  question? 

(How  did  we  get  to  the  moon  without  finite  element  codes?) 

Don’t  be  afraid  to  consider  the  possibility  that  there  is 
no  discernable  cause/effect  relationship 
in  what  you’re  trying  to  simulate. 
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Sometimes,  it’s  better  to  be 
lucky  than  good... 


...but  don’t  count  on  it! 


“Noah  had  an  absurd  idea  that  he 
could  navigate  without  any  knowledge 
of  navigation,  and  he  ran  into  the  only 
shoal  place  on  earth.” 

-Mark  Twain 
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Stat-€cis© 


statistics  made  easy- 


How  to  Frame  a  Robust  Sweet  Spot  via 
Response  Surface  Methods  (RSM) 


Frame  a  Robust  Sweet  Spot 


By  Mark  J.  Anderson,  PE,  CQE 
Stat-Ease,  Inc.,  Minneapolis,  MN 

mark@statease.com  612-746-2032 


Stcit-€cis© 


statistics  made  easy- 


Strategy  of  Experimentation 


Frame  a  Robust  Sweet  Spot 
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1. 

2. 

3. 


Response  Surface  Methods  (RSM)* 

When  to  Apply  It  (Strategy  of  Experimentation) 


Fractional  factorials  for  screening 

High-resolution  fractional  or  full  factorial  to 
understand  interactions  (add  center  points  at  this 
stage  to  test  for  curvature) 

Response  surface  methods  (RSM)  to  optimize. 

Contour  maps  (2D)  and  3D 


1. 

2. 

3. 


surfaces  guide  you  to  the  peak. 


Frame  a  Robust  Sweet  Spot 


3 


RSM:  When  to  Apply  It 


Stcit-€cis© 

statistics  mod®  easy” 


Use  factorial  design  to 
get  close  to  the  peak. 
Then  RSM  to  climb  it. 


Region  of  Interest 


Region  of  Operability 


Frame  a  Robust  Sweet  Spot 


RSM  vs  OFAT 
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statistics  made  easy 


ill 
C 0 


or 


Frame  a  Robust  Sweet  Spot 


FactorA 
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RSM:  Process  Flowchart 


Stcit-€cis© 

statistics  mod®  easy 


Subject  Matter  Knowledge 
(Plus  Factorial  Screening) 

Vital  Few  Factors  (x's) 


Process 


->  Measured  Response(s)  (y(s)) 


u^irJihiiiilihiihliiiiihLMBB 

Fitting*  = 

i^^^||||[|l||i)||||i||l||i|i|i||||l^|^ 


Polynomial  Model 

Response  Surface 


"All  models  are  wrong ,  but  some  are  useful "  -  George  Box 

Frame  a  Robust  Sweet  Spot 


Case  Study  -  RSM  Design  &  Analysis 


Aerospace  Example 


Via  a  face-centered  central  composite  design  (FCD)  aimed  at 
minimizing  weight  of  an  active  aeroelastic  wing,  aerospace 
engineers  studied  three  vital  structural  factors: 

A.  Aspect  ratio,  3-5. 

B.  Taper  ratio,  0. 2-0.4. 


C.  Thickness  ratio,  0.03-0.06 


"A  designer  knows  he  has  achieved  perfection 
not  when  there  is  nothing  left  to  add, 


but  when  there  is  nothing  left  to  take  away  " 


-  Antoine  de  Saint-Exupery 


*(RSM  Simplified:  Optimizing  Processes  Using  Response  Surface  Methods  for  Design  of  Experiments, 
Mark  J.  Anderson  &  Patrick  J.  Whitcomb,  Productivity  Press,  NY,  NY  (2007)  Chapter  10,  pp:  224-228.) 


Frame  a  Robust  Sweet  Spot 
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Response  Surface  Map  for  Wing  Weight 


Stcit-€cise 


statistics  mode  ©as 


The  picture  tells  the 
story.  It's  generated  by 
the  fitted  -equation 
(math  model),  which 
also  provides  a  "transfer 
function"  for  numerical 
prediction  and 
optimization. 


O) 

CD 


CD 

c 


0.03 


A:  Aspect 


Equation  in  Terms  of  Coded  Factors: 

Log10(Wing  Weight)  =  +2.56  +  0.19  A  +  0.037  B-0.21C 


Equation  in  Terms  of  Actual  Factors: 
Log10(Wing  Weight)  = 

+2.29660 

+0.19251  *  Aspect 

+0.37457  *  Taper 

-13.86641  *  Thickness 


Factors  Tool 


|  Gauges  Sheet 


fxT  A:Aspect 


>12:  C:  Thickness 


0.20 


Term 


3.00  0.06 


0.06 


C:  Thickness 


Frame  a  Robust  Sweet  Spot 


Data  file:  Wing  weight 
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Stcit-€cis© 

statistics  mod®  easy 


Graphical  Optimization  of  Multiple  Responses 

to  Generate  Design  Space 


By  overlaying  contour  plots  for  multiple  responses  - 
shading  out  regions  out  of  spec,  one  can  view  the 
design  space  (aka  "operating  window"  or  "sweet 
spot").  The  FDA  defines  "design  space"  as  the 
"multidimensional  combination  and  interaction  of 
material  attributes  and  process  parameters  that  have 
demonstrated  to  provide  assurance  of  quality."  This  is 
a  key  element  for  their  quality  by  design  (QbD) 
initiative.  It  merits  attention  for  test  and  evaluation. 


Frame  a  Robust  Sweet  Spot 
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C:  Power 


Stcit-€cis© 

stotistics  mad®  easy 


Simple  Example  of  Design  Space 

Making  Microwave  Popcorn  (1/2) 


Try  this  experiment  at  home!  Where  is  the  "sweet  spot" for 
making  popcorn?  (Hint:  Want  low  unpopped  kernels  -  UPK 
-  and  high  taste  rating.) 


Taste  UPKs 


4.00  4.50  5.00  5.50  6.00  4.00  4.50  5.00  5.50  6.00 

B:Time  B:  Time 


Frame  a  Robust  Sweet  Spot 


Data  file:  Popcorn 
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Simple  Example  of  Design  Space 

Making  Microwave  Popcorn  (1/2) 


Stcit-€cis© 

stotistics  mad®  easy 


This  is  the 
"sweet  spot 
for  making 
popcorn. 


Frame  a  Robust  Sweet  Spot 


B:  Time 
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Stcit-€cis© 

statistics  mod®  easy” 


Case  Study  -  Design  Space 
Aerospace  Example* 


Via  an  optimal  RSM  design  aimed  at  characterizing 
a  freejet  nozzle's  exit  profile,  aerospace  engineers  studied 
two  vital  factors: 

A.  Temperature,  low  to  high. 

B.  Pressure,  low  to  high. 


Over  an  area  of  interest  that  required  a  linear  constraint  to 
cut  off  the  region  where  both  factors  hit  their  high  levels. 
The  actual  levels  tested  remain  confidential.  However, 
facility  support  testing  at  temperatures  up  to  4,700  degrees 
Rankine  and  pressures  up  to  2,800  psia. 


*("Developing,  Optimizing  and  Executing  Improved  Test  Matrices"  presented  by  Dusty  Vaughn  and 
Doug  Garrard  to  the  U.S.  Air  Force  T&E  Days  2009,  approved  by  U.  S.  Government  for  public  release  via 
the  American  Institute  of  Aeronautics  and  Astronautics.) 

Frame  a  Robust  Sweet  Spot  i: 


Stcit-€cise 


statistics  mode  eosv“ 


Defining  the  Operating  Constraints 


This  is  a  "burnt  pudding"  problem  -  too  much  temperature  and 
time  overcooks  the  food.  DOE  software  makes  it  easy  to  avoid 
these  unwanted  combinations.  The  experimenter  need  only 
identify  the  constraint  points. 

Here,  after  entering  dummy  values  for  each  factor,  a  constraint 
point  is  set  for  the  level  of  temperature  that  cannot  be  exceeded 
when  the  system  is  at  high  pressure. 

Conversely,  a  second  constraint  point  is  set  for  the  maximum 
pressure  level  when  temperature  is  at  its  highest  level. 


Name 

Low  Actual 

High  Actual 

Vertex 

<  >  skip 

Constraint  Point 

A: 

Temperature 

3000 

4000 

4000 

A< 

3150 

B: 

Pressure 

1000 

2000 

2000 

B  < 

1500 

Frame  a  Robust  Sweet  Spot 
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?l9,t-„fQs©  Laying  Out  an  Optimal  Design 


Due  to  the  demands  of  cost  and  schedule,  the  experimenters 
chose  a  minimum-run  design  of  6  points  to  fit  the  standard  second- 
order  (quadratic)  RSM  model.  One  point  was  replicated. 


However,  for  expository 
purposes,  here  is  a  stouter 
design*  with  4  additional  test 
points  to  assess  lack-of-fit  and 
4  points  replicated  for  a 
stronger  estimate  of  pure  error. 
Also,  the  optimality  criterion 
for  this  design  is  IV  -  now 
favored  for  RSM  designs,  not  D- 
optimal  as  done  by  the 
experimenters. 


A:  Temperature 


*(How  many  test  points  will  be  needed  is  an  issue  of  power,  which  goes  beyond  the 
scope  of  this  talk.  For  details  on  design-sizing  for  RSM ,  see  the  Sept.  '08  Stat-Teaser.j 

Frame  a  Robust  Sweet  Spot 
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Stcit-€cis© 

statistics  mod®  easy 


Results 


The  following  response  surfaces  were  generated  via  re-simulation  from 
predictive  equations  provided  in  coded  form  by  the  experimenters.  The 
graphs  closely  resemble  the  published  results  for  the  key  measures  of 
dynamic  pressure  (Q)  and  total  sensible  enthalpy  (Hts). 


Frame  a  Robust  Sweet  Spot 


Stcit-€cis© 

statistics  mod®  easy” 


Sweet  Spot  (Hypothetical) 


The  customer  requirements  have  not  been  revealed,  but  assume 
they  are  represented  by  the  graphical  overlay  shown  below. 


Frame  a  Robust  Sweet  Spot 


A:  Temperature 
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Stcit-€cis© 

stotistics  mad®  easy 


Robust  Sweet  Spot 


To  be  more  conservative 
(robust)  in  framing  the 
sweet  spot,  superimpose 
the  confidence  intervals 
(Cl)  -  a  function  of  the 
underlying  standard 
deviation  (provided  by 
the  original  publication) 
and  the  power  of  the 
experiment  design 
(stronger  in  our  re¬ 
simulation).  The  flag  in 
the  center  might  mark  a 

good  place  to  operate! 


i.oo  — 


c n 


Q: 

1495.473 

Hts: 

698.972 

XI 

3522.70 

X2 

1300.97 

Hts  Cl:  750.000]  I  Hts:  750.0001 


0 


1300.1 


CL 

oc i 


Hts:  650.000 


I  Hts  Cl:  650.000] 

>3 


1200. 


1100. 


3375.00 


3450.00 


3525.00 


3600.00 


3675.00 


A:  Temperature 


Frame  a  Robust  Sweet  Spot 
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Conclusion 


Stcit-€cis© 

stotistics  mad®  easy 


Via  application  of  response  surface  methods  (RSM) 
experimenters  in  the  field  of  test  and  evaluation 
can  frame  an  operating  window  (aka  "sweet  spot" 
or  "design  space").  To  be  more  conservative 
(robust),  shade  out  the  regions  that  fall  within  the 
confidence  intervals  of  the  boundary  lines. 


Frame  a  Robust  Sweet  Spot 
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Stcit-€cis© 

stotistics  mad®  easy 


Frame  a  Robust  Sweet  Spot 


Statistics  Made  Easy® 


Best  of  luck  for  your 
experimenting! 

Thanks  for  listening! 

-  Mark 

Mark  J.  Anderson,  PE,  CQE 
Stat-Ease,  Inc. 
mark@statease.com 
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Relationship  within  TRMC 
Synergy  through  Aligned  Investment 


Service 
T&E/S&T 
Working  Groups 


Quadrennial  Defense  Review 
Strategic  Planning  Guidance 


DoD  Strategic  Plan 
for  T&E  Resources 


i 
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Service 

T&E  Needs  and 
Solutions  Process 


TRMC 

Joint 

Investment 

Programs 


Risk  mitigation  needs 
Technology  shortfalls 


Risk  mitigation  solutions 
Advanced  development 


Service  Improvement 
&  Modernization/ 
Programs 


Acquisition  Programs  / 
Advanced  Concept 
Technology  Demonstrations 


T&E  Multi-Service 
/  Agency 
Capabilities 


DoD  Corporate 
Distributed  Test 
Capability 


What  is  Distributed  Testing? 


A  process,  preferably  persistent  and  continuous,  for  linking 
various  geographically  separated  Live,  Virtual,  and 
Constructive  sites  and  capabilities  together  in  a  distributed 
environment,  for  use  across  the  acquisition  life  cycle,  to 
support  and  conduct  the  Test  and  Evaluation  (T&E)  of  a 

system  or  systems-of-systems. 


GOAL:  Near  Real-Time  Test-Fix-Test 

_ 
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What  is  Distributed  Testing? 


A  process,  preferably  persistent  and  continuous,  for  linking 
various  geographically  separated  Live,  Virtual,  and 
Constructive  sites  and  capabilities  together  in  a  distributed 
environment,  for  use  across  the  acquisition  life  cycle,  to 
support  and  conduct  the  Test  and  Evaluation  (T&E)  of  a 

system  or  systems-of-systems. 


A  new  way  of  thinking  for  many  in  the 

Test  and  Evaluation  enterprise 

_ 
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Distributed  Testing  Impacts 


•  Distributed  Testing  has  already  demonstrated: 

•  Time  savings,  risk  reduction,  cost  savings 

•  Efficiencies  across  the  development  and  T&E  process 

•  Early  identification  of  issues 

•  Move  data — not  people 

•  Near  real-time  Test-Fix-Test 

•  Distributed  Testing,  when  fully  implemented  also: 

•  Provides  for  agile,  persistent  T&E 

•  Supports  early  integration  of  DT  and  OT 

•  Gives  SME’s  an  — Iralnsive  Lab”  and  connective  relationship  with 
other  entities  in  the  systems-of-systems  environment  that  they 
wouldn’t  have  otherwise. 
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Why  Consider  Distributed  Test? 


•  Do  you  have  data  exchange  requirements  within  your 
system  or  within  a  system-of-systems  (SoS)? 

•  Do  you  have  a  requirement  to  address  SoS  interoperability 
issues  early  in  the  acquisition  process? 

•  Do  you  have  adequate  numbers  of  systems  under  test  for 
live  testing? 

•  Do  you  have  adequate  numbers  of  — suporting  cast” 
(supporting  systems,  C4ISR  assets,  etc.)  for  live  testing? 

•  Do  you  have  adequate  threat  types,  fidelity  and  density  in 
realistic  numbers  at  realistic  ranges  for  live  testing? 


7 


When  Is  Distributed  Test  Appropriate? 


Examples  Where  Appropriate 


-  Interoperability  testing 

•  C4  Interoperability  with  higher,  lower,  and  adjacent  Joint  force  organizations 

-  Data  exchange  in  early  DT  testing 

•  Interaction  between  sub-systems  (latency  may  be  a  consideration) 

•  Interaction  between  systems  in  a  realistic  environment 

o  Provide  the  most  realistic  environment  possible  from  concept  exploration  through 
Follow-On  T&E 


-  When  it  is  too  costly  to  bring  all  the  player  systems  together  on  a  single 
range 

-  Gain  or  increase  operational  relevance 

•  Virtual  and  Constructive  capabilities  to  supplement  live  systems  (e.g.,  red 
forces,  white  forces,  terrain,  immobile  test  assets) 

•  Examples  Where  Inappropriate 

-  System  performance  tests  that  do  not  include  other  systems/subsystems 

-  Reliability  testing 

Reduces  Cost,  Risk,  and  Time  I 
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Distributed  Testing  Challenges 


Not  unique  to  JMETC,  but  we  are  working: 

•  Classification 


-  Multi-level  security  issue  to  peer  to  networks  of  higher  classification  levels 

-  Solution 

•  Short  Term:  Create  separate  enclaves  for  each  level 

-  Time  and  dollars  issue  to  operate  at  higher  levels  of  classification 

•  Long  Term:  Develop  an  enterprise  solution 

-  Current  CTEIP  Project 


•  DOD  Information  Assurance  Certification  and  Accreditation  Process 

-  Information  Assurance  Requirements  for  higher  levels  of  classification 
•  Time  and  dollars  issue 
-  DIACAP  Tiger  Team 

•  Common  lexicon  and  reciprocal  acceptance 

•  RDT&E  Community  won  a  mechanism  for  their  voice  to  be  heard  by  the  policy 
makers 

•  TRMC  is  now  a  non-voting  member  of  the  DIACAP  Technical  Advisory  Group  (TAG), 
where  next-generation  policy  is  being  developed 
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The  JMETC  Mission 


JMETC  provides  the  persistent  and  robust 
infrastructure  (network,  integration 
software,  tools,  reuse  repository)  and 
technical  expertise  to  integrate  live,  virtual, 
and  constructive  systems  for  test  and 
evaluation  in  a  Joint  Systems-of-Systems 

environment 
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What  is  JMETC? 


•  DoD  enterprise  approach  for  linking  distributed  facilities 
currently  being  used  by  over  60  test  facilities 

*  A  core,  reusable,  and  easily  reconfigurable  infrastructure 

*  Consists  of  the  following  products: 

•  Persistent  connectivity 

•  Middleware 

•  Standard  interface  definitions  and  software  algorithms 

•  Distributed  test  support  tools 

•  Data  management  solutions 

•  Reuse  repository 

•  Provides  customer  support  team  for  JMETC  products 
and  distributed  Live,  Virtual  &  Constructive  DT  and  OT 
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JMETC  Enabled 
Distributed  Testing 


Joint  Operational  Scenarios 


Systems 

Under 

Test 

Integrated 

Test 

Resources 


Virtual 

Prototype 


TENA 

Standard 

Interface 

Definitions 


TENA 

Common 

Middleware 


Hardware 
in  the 
Loop 
Lab 


TENA 

Standard 

Interface 

Definitions 


TENA 

Common 

Middleware 


JMETC 
Network  on 
SDREN 


Reuse 

Repository 


JMETC 
Infrastructure 


Installed 

Systems 

Test 

Facility 


TENA 

Standard 

Interface 

Definitions 


TENA 

Common 

Middleware 


Range 


TENA 

Standard 

Interface 

Definitions 


TENA 

Common 

Middleware 


Distributed  Test 
Support  Tools 


Environment 

Generator 


TENA 

Standard 

Interface 

Definitions 


TENA 

Common 

Middleware 


Threat 

Systems 


TENA 

Standard 

Interface 

Definitions 


TENA 

Common 

Middleware 


Data  Management 
Solutions 


VVPAFB:  SIMAF 


Boeing:  Berkeley 


Nellis  AFB:  CAOC-N 


Redstone|6):  RTC:  DTCC.  GMAN 
SED:  Patriot.  THAAD.  FAAD,  SIV 


Tinker  AFB:  AWACS 


SMDC  Cliailestol 
s  IPC.MEFil 


i  Redstone 


Ft  Htiachlica:  JTRS  WSMRIRCC 


Greenville:  Rivet  Joint 


Melbourne: 
NGC  JSTARS 


Boeing:  Tukwila 
eyport:  NUWC 


_ 


Ft  Lewis:  EPG 


Site  in  Alaska 
AFt.Gieely:  CRTC 


~7: 


» C  ampteiuTleton: 


Sites  in  SoCal 
OEdwards:  Ridley 

•China  Lake  (3i: 
AV-8B.F  A- 18. 
IBAR 

•Point  Mikjii  (2): 

ITEC.EA-6B 
OEI  Seyundo: 

‘Bi 

MCTSSA 
▲Coiona:  NSW: 

•  Point  Loina  <2i: 
RLBTS 

SSC-PAC 

A  SSC-PAC  HAIPE 
A  Rancho  Bernardo. 
^NGC  BAMS 
^fewest  Ayg  Rti. 


Sites  in  Hawaii 
•PMRF:  Bldg  105 
•  MHPCC 


JMETC  Connectivity 

O  Functional  Sites:  60 
A  New  Sites  Planned:  9 
'fc  Connection  Points  to  Other  Networks:  4 

Dedicated,  trusted  connectivity  on  SDREN  (part  of  the  GIG) 

Encrypted  for  Secret  -  System  High 

DISA-registered  IP  address  space 

Active  monitoring  of  network  performance 

Capable  of  supporting  multiple  simultaneous  test  events 


rm 


ay  Proving  Ground 


Schriever  AFB:  JEWL 

o 


Ft  Hiiacliuca:  (2)  JITC,  JTDL 


Ft  Hood  12):  CTSF,  TTEC 


Sites  in  Gulf  Range 
OHuilhuit  Field:  C2DAC 

OEglin  AFB  (4|: 

AOC.DTF.  G\AEF. 
KHILS 


□  Sues 


I  01 


i  n 


C  HR 


As  of  14  Feb  2011 


Hanscom  AFB:  CEIF 

PCX' 

Betlipage:  NGC  BAMS 
Monmouth:  JOIN 

Sites  in  MD.  DC.  VA 
Aheideen:  ACCN 
Pax  River  |5i: 
•ESTELE2C  D.  MCL 
^  ACETEF,  SAIL.  ATR 
•  JMETC  SYSCON 
XEast  Ayg  Rti. 

iSj  OPentagon:  WARCAP 
•DISA:  Sky  7 
#Dahlyren<2>:  CEDL.IWSL 
•JFCOM:  JSIC 

O  Langley  <2>: 

C-GIIF.  TDLITC 
•Noifolk:  COMOPTEVFOR 
•Dam  Neck:  CDSA 

•Wallops  Island: 

(2ISCSC.SSDS 
ANewpoitNews: 

NGC  VASCIC 
AMcLean:  MITRE  NCEL 


©  Aimy 
O  Ail  Force 

♦  Navy 

•  Marines 

#  Joint 
O  Industry 


JMETC  Allows  You  to  “Test  Early  and  Test 
Often”  Across  the  Acquisition  Life  Cycle 


Outline  Distributed 
Testing  requirements 
in  the  TEMP 


Concept  Component 
Exploration  Advanced 
Development 

Decision 
▼  Review 

Concept  &  Tech  Development 


Pre-Systems  Acquisition 

▲  A 

Lk 


Enables  early  verification  that 
systems  work  stand  alone  and  in 
a  Joint  Environment 


Support  to  Acquisition  Programs 
with  the  expertise  to  integrate 
distributed  test  facilities 


Rapid  Acquisition,  Developmental  Test,  Operational 
Test,  Interoperability  Certification,  Net-Ready  Key 
Performance  Parameters  testing,  Joint  Mission 
Capability  Portfolio  testing 


IOC 


System 

Integration 


Interim 
Progress 


Review 


X\ 


LRIP  Full-Rate  Production 
&  Deployment 

FRP 

♦  Decision 
Review 

Production  &  Deployment 


Operations 
&  Support 


Systems  Acquisition 


Sustainment 


Helps  find  problems  early  in 
acquisition  -  when  they  are  less 
costly  to  fix 


(Engineering  &  manufacturing  development, 
demonstration,  LRIP  &  production) 

^ 


/\ 


JMETC  enables 
testing  across  the 
acquisition  life  cycle 

JMETC  can 
potentially  reduce 
test  time  and  cost 


By  Providing 


•  Readily-available, 
persistent  connectivity 
with  standing  network 
security  agreements 

•  Common  integration 
software  for  linking  sites 

•  Accredited  test  tools 
for  distributed  testing 
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FY10  JMETC  Events 


Customer 

Event 

Record  Event  Dates* 

Air  Force 

JEFX  10-1/Spirit  Ice  (B2  Data  Link  Test) 

October  -  November  2009 

Navy 

BAMS  LVC  DE 

October  -  December  2009 

Air  Force 

Battlefield  Airborne  Comm.  Node  (BACN)  JUON 
(DT/OT) 

November  2009  to  September  2010 

Air  Force 

Agile  Fire  10-2 

January  2010 

Air  Force 

JEFX  10-2/3 

February/April  2010 

Joint 

Joint  Surface  Warfare  JCTD 

February  to  September  2010 

JS  J8/JIAMDO 

Joint  Sensor  Integration 

April  to  September  2010 

Air  Force 

B1-B  Fully  Integrated  Data  Link  Testing 

April  2010 

JFCOM  J84/89  (TEST  WEEK) 

JCAS  Distributed  Test 

June  2010 

JIAMDO  (Navy  Lead) 

Correlation/Decorrelation  Interoperability  Test 
(C/DIT)  Integration  Events  (Continuous) 

July  to  September  201 0 

Army  (Lead) 

UAS  in  the  National  Airspace 

July  to  September  201 0 

Air  Force 

Agile  Fire  10-3 

August  2010 

Joint 

JITC  Joint  Interoperability  Tests 

JIT  10-3  &  11-1 

Discussions  for  Future  Teaming 

Gerald  R.  Ford  Class  (CVN-78) 

Joint  Strike  Fighter  (JSF) 

JIAMDO/Joint  Track  Manager 

Brigade  Combat  Team  (BCT) 
Modernization 

Multi-Function  Adv  Data  Link  (MADL) 

Multi-Mission  Maritime  Aircraft  (MMA) 

* 


% 


Each  event  is  normally  preceded  by  1-3  spirals:  Connectivity  Check,  Integration,  and  Dry  Run 


JMETC  FY10  Accomplishments 


Joint  Integrated  Air  &  Missile  Defense 
Organization  (JIAMDO) 

Broad  Area  Communications  Node  (BACN)  JUON 
B1-B 

Broad  Area  Maritime  Surveillance  System  (BAMS) 
Air-Ground  Integrated  Layer  Exploration  (AGILE) 
Joint  Interoperability  Test  Command  (JITC) 


JMETC  Accomplishments 


Supported  88  distinct  customer  test  activities 

Expanded  network  from  38  to  57  sites 

ATIN  and  JTDL  Networks  transitioned  to  JMETC 

Upgraded  JMETC  support  applications  and  utilities 
to  TENA  R6 

DIACAP  Tiger  Team  report  completed  and 
recommendations  being  executed 

Enhanced  JMETC  services  and  capabilities 
provided  by  leveraging  InterTEC,  Services,  and 
Industry 

Reuse  Repository  usability  improvements 


Selected  Benefits  to  the  DoD 


Integrated  DT  &  OT  on  a  Joint  Urgent 
Operational  Need  for  the  warfighter 

Maximized  usage  of  theater  assets 
during  limited  maintenance  windows 

Improved  Joint  track  information  sharing 
to  ensure  interoperability  of  systems  in 
theater  operations 

Coalition  exchange  and  examination  of 
real-time  air  picture  data 

Identification  of  Air  Force  Initiatives 
ready  for  warfighter  transition 

Investigated  tactical  UAS  deployment  in 
the  National  Airspace 

Employment  of  Net-Enabled  Weapons 

JCAS  immediate  request  &  end-to-end 
processes  —ass”  characterization 

Determined  distributed  system 
components  were  not  ready  for  full  live 
integration  testing 

Executed  testing  to  support  system-of- 
system  interoperability  certification 


FY11  JMETC  Events 
(More  to  Come) 


Customer 

Event 

Record  Event  Dates* 

Joint 

JIAMDO  Correlation/Decorrelation  Interoperability  Test  (C/DIT) 
Integration  Events  (Continuous)  OCONUS 

October  2010 

Navy 

UAS  in  NAS  Runs  for  Record 

October  2010 

Internal 

InterTEC  Spiral  3  Systems  Acceptance  Test 

October  -  November  2010 

Joint 

JITC  Joint  Interoperability  Tests  JIT  11-1,2,3,4,5  (Continuous) 

October  201 0  -  September  201 1 

Air  Force 

B1-B  Fully  Integrated  Data  Link  Testing 

November  2010 

Air  Force 

AGILE  Fire  Phase  III  /  JEFX  2011 

December  201 0  -  February  201 1 

Joint 

JTRS  JPO  --  JTRS  Ground  Mobile  Radio 

January  2011 

Navy 

Broad  Area  Maritime  Surveillance  (BAMS)  (Continuous) 

January  -  September  201 1 

Joint 

Joint  Track  Manager  Capability  Demonstration  (Continuous) 

January  -  September  2011 

Joint 

JSF  Initial  M&S  Interoperability  (Continuous) 

February  -  March  2011 

Air  Force 

JSTARS  Interoperability  Test 

May  2011 

Joint 

JS  J8/JIAMDO  Joint  Sensor  Integration 

June  -  August  2011 

Discussions  for  Future  Teaming 

Gerald  R.  Ford  Class  (CVN-78) 

Global  Hawk 

GATOR 

Brigade  Combat  Team  (BCT) 
Modernization 

F-22  FY12  Testing  Planned 

Multi-Mission  Maritime  Aircraft  (MMA) 

Each  event  is  normally  preceded  by  1-3  spirals:  Connectivity  Check,  Integration,  and  Dry  Run 
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JMETC  Testi 


B-2  Spirit  ICE  Data  Link 
Test  (Nov  2009) 


JEFX  assessment  of  B 


Link- 1 6  interoperability 
challenges  with  AW  ACS 


Connected  live  B-2  on  ramp 
at  Whiteman  AFB,  MO,  an 
AWACS  HITL  at  Tinker 
AFB,  OK,  within  a 
distributed  C2  environment 


T— Cost  Savings  and  Better 


As  of  14  Sep  2009 


Product! 


Marines 

Joint 

Industry 


•Early  testing  led  to  early  identification 
•  Time  sensitive  targeting  and  correction  of  Link  1 6 
scenarios  with  combat  ready  interoperability  issues 
crews 


•No  range  or  flying  costs! 
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JMETC  Testing  Success 


Joint  Surface  Warfare  JCTD 

•  Point  Mugu  Test  Team 
demonstrated  Net  Enabled 
Weapon  Link- 16  capability 
using  F/A- 1 8E/F  as  launch 
platform,  JSOW  C-l  as 
weapon,  and  JSTARS  as  3rd 
party  target  source 

•  Distributed  Tests 

•  09-11  Mar  2010 

•  04-05  May  2010 

•  17-19  Jun  2010 

•  31  Aug -01  Sept  2010 


Melbourne: 
ARS 


IMPACT— Efficiency,  Lower 
Technical  Risk,  and  Cost  Savings! 

Program  scheduled  &  executed  short 
multiple  tests  for  incremental  software 
update  evaluation 

Resources  expended  on  test  &  analysis 
and  not  network  setup  and  monitoring  „ 


JMETC  Testing  Success 


Joint  Integrated  Air  and 
Missile  Defense 
Organization  (JIAMDO) 

•  Correlation/Decorrelation 
Integrated  Test  (C/DIT-10) 
to  improve  interoperability 
of  Aegis,  E2C,  CAC2S, 
AW  ACS,  Patriot,  and 

FA  AD. 

•  During  Oct  2010  testing, 
JMETC  enabled  multiple 
C/DIT  runs  with  an 
average  turnaround  time  of 
1 1  minutes  -  two  shifts  per 
day 


IMPACT— Efficiency ! 

•  Near  real  time  test-fix-test 

•  C/DIT  FY-11  T&E  events  accelerated 
into  FY 1 0,  w/no  funding  impacts  to  FY- 
10 
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Interoperability  Certification 


Joint  Interoperability  Test 
Command  (JITC):  Joint 
Tactical  Data  Link  (JTDL) 
Testing 

JITC  conducts  interoperability 
assessments,  standards 
conformance  and 
interoperability  certification 
testing  of  joint  tactical  data  links 
in  HWIL  and  operationally 
realistic  environments  to 
validate  the  implementation  of 
approved  standards  in  a  Joint 
environment. 

JITC  uses  JMETC  Connectivity 
and  tools  for  JTDL  Testing 


IMPACT— Test  Commonality! 

JITC  Interoperability  Certification 
is  required  for  Net  Ready  KPP  for 
all  AC  AT  Programs 

JITs  use  JMETC  infrastructure. 


(Afanned.  Unmanned,  Space-based,  National  and 
Procesang/Dissemination  Colters) 


C  VN  78  & 


^ffigber  Author  ity 

(OS/A  OGA, 
SECDEF/JCS, 
JFC/JF  Component 
Commanders) 


Land  Attack  Forces 
(including  Strike  Forces, 
Expeditionary  Forces, 
Land  Forces,  and  SOP) 
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Broad  Area  Maritime  Surveillance 
Unmanned  Aircraft  System  (BAMS  UAS) 


DAMS  PAYLOAD  DATA  -  AtDDORNC  COMMUNICATIONS  RELAY 

DAMS  AR  VDIICLC  C2  DAMS  UAS  NODE  (DAMS  UA  A  DAMS  MOO.TOD) 


-  -  -  -  DEFENSE  INf  OWMAIIOH  SYSTEMS  NETWORK  >  GLOOAL  INFORMATION  GRID 


Program  Status/Events: 

BAMS  planned  sites  are:  Bethpage  -  NGC  MSSIL 
(existing),  Rancho  Bernardo  -  (Installing),  Dam  Neck 
C2/S A/TCC/MOC  (existing),  Palmdale  NGC  SIL 
(TBD),  NAS  Patuxent  River  (existing) 

Current  BAMS  schedule:  June  2012  (6-12  months) 
NGC  lead.  June  20 1 3  -  IOC  Pax  Lead 

PSP  signed  by  BAMS  and  JMETC  August  6,  2010 


System  Architect  v  10.1.11  Encyclopedia  BAMS_PBSS  (11  Jan07)v1.1 


Issues: 


Program  Description: 

BAMS  UAS  is  an  integrated  Systems  of  System 
that  will  provide  multi-sensor  persistent 
maritime  ISR  to  the  Maritime  Patrol  and 
Reconnaissance  Force 

Program  PQC: 

Jeff  Sappington  NAVAIR 


Working  to  peer  with  BAMS  Classified  Network  (BCN)  but 
may  be  separate  agreement  with  NGC 

ESP  for  flight  test  needs  to  be  completed,  ESP  format 
changes  under  review  by  ENG/DOPS 

Both  BAMS  and  NGC  are  still  discovering  potential  T&E 
requirements  including  various  networks  that  BAMS 
interfaces  with  for  flight 

Last  Contact: 


BAMS  Technical  Exchange  Meeting  Rancho  Bernardo,  CA 
March  1-3,2011 


22 


JMETC  Users  Group 


•  Purpose  is  to  focus  on  technical  requirements  and  solutions  relevant 
to  current  and  future  Distributed  Testing  needs. 

•  Technical  and  Management  level  representatives  identify  core  infrastructure 
requirements,  and  most  importantly  resolve  issues 

•  Quarterly  meetings  of  250-300  JMETC  customers,  acquisition 
programs,  test  events,  ranges,  LVC  sites,  tools  and  network  providers 


An  established  forum  for  the 
Distributed  Test  Community  to: 

o  Identify  core  infrastructure 
requirements  and  use  cases 

o  Identify,  investigate,  &  resolve 
issues 

o  Identify  opportunities  to 
collaborate 

o  Discuss  available  solutions, 
tools,  and  techniques 

o  Share  lessons  learned 


Next  JMETC  Users  Group  Meeting: 

•  Scheduled  for  March  22-23,  2011 

•  Location:  Norfolk,  VA 

•  Tracks: 

•  User  Requirements 

•  Information  Assurance  / 
Security 

•  Data  Management 

•  InterTEC  (Current  &  Planned) 

•  Networking 
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Summary 


•  Distributed  Testing  can  save  Acquisition  T&E  Programs  time 
and  money  and  result  in  better,  more  interoperable  products 
while  reducing  technical  risk! 


•  JMETC  is  here  and  proven! 

•  Many  Sites  and  Systems  already  connected  via  JMETC  and  well 
versed  in  TENA  and  the  InterTEC  tools 

•  Demonstrated  reliability  with  the  capability  to  execute  multiple  events 
simultaneously,  supporting  high  data  rates  and  low  latency 
requirements 

•  Multiple  examples  of  JMETC  value  added  for  customers 

•  Provides  Acquisition  T&E  Programs  near  real-time  Test-Fix-Test 
capability 

•  JMETC  offers  support  to  develop  our  customer’s  distributed  test 
requirements 


You  need  only  contact  us 
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JMETC  Program  Points  of  Contact 


JMETC  Program  Manager: 

JMETC  Principal  Deputy  PM: 
JMETC  Lead  Operations  Planning: 
JMETC  Senior  Technical  Advisor: 
JMETC  Lead  Systems  Engineer: 

JMETC  Lead  Network  Engineer: 


Chip  Ferguson  chip.ferquson@osd.mil 
703-601-5274 

Bruce  Bailey  bruce.bailev@osd.mil 
703-601-5208 

Marty  Arnwine  martemas.arnwine@osd.mil 
703-601-5215 

George  Rumford  qeorqe.rumford@osd.mil 
703-601-5233 

Ryan  Norman  rvan.norman@osd.mil 
703-601-5277 

Arjuna  “AJ”  Pathmanathan 

Ariuna.Pathmanathan@osd.mil 

703-601-5214 


JMETC  Website:  www.imetc.org 
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Backup  Slides 


Ft  Lewis: 
EPG 


Sites  in  Hawaii 
APMRF:  Bldg  105 
AMHPCC 


How  an  Air  Force  Customer  Sees 
Distributed  Connectivity 


Hardware-in-the-Loop  (HWIL)  Availability 


w 


Hanscom  AFB:  CEIF 
Bethpage:  NG  BAMS 
t.  Monmouth:  JOIN 


Nellis  AFB:  CAOC-N/ASOC 

.Cl 


Whiteman:  B-2  Q 


I  _ Tinker  AFB: 

Kirtland  AFB:  AWACS  O 

QSDOCC  Q  w 


Boeirrg-St.  Louis: 
CIDS 


WSMR:  IRCC 


Greenville:  Rivet  Joint 

A 


Ft  Huachuca:  JITC 


Ft.  Worth:  AFEWES 

O 

Ft  Hood  (2):  CTSF,  TTEC 


All  linked  by  JMETC 


one  (3):  DTCC,  GMANX3ED 

Charle^n-(2): 
IPC,  MEF-MEU 


Melbourne:  JSTARS 


O  Army 
O  Air  Force 
O  Navy 
•  Marines 
O  Joint 
O  Industry  ^ j 


JMETC  Planning  With  20+  Customers 
(Active  and  Prospective) 


Joint  Tactical  Data  Link  (JITC  JTDL) 

Joint  Expeditionary  Forces  Experiment  (JEFX) 

Joint  Integrated  Air  and  Missile  Defense 
Organization  Corr/Decorr  Interoperability  Test 
and  Joint  Sensor  Integration  (JIAMDO  C/DIT  & 
JSI) 

Air-to-Ground  Integrated  Layer  Exploration 
(AGILE  Phase  III  and  IV) 

Network  Enabled  Weapons  Interoperability 
Working  Group  (NEW  IWG) 

Unmanned  Aircraft  Systems  in  National 
Airspace  (UAS  in  NAS) 

Digitally  Aided  Close  Air  Support  (DACAS) 

Space  Threat  Assessment  Testbed  (STAT) 

Joint  Unmanned  Aircraft  Systems  Mission 
Environment  (JUAS  ME) 

Joint  UAS  Digital  Information  Exchange 
(JUDIE)  Joint  Test  and  Evaluation  Program 


Acquisition  Programs/PEOs 


Joint  Strike  Fighter 

F-22  Block  3.2  Link  16  Receive  Testing 
Multi-Function  Advanced  Datalink  (MADL) 
Battlefield  Airborne  Communications  Node 
(BACN)  Joint  Urgent  Operational  Need 
B-1  Fully  Integrated  Data  Link  (FIDL) 

PEO  Integrated  Weapons  Systems 
CVN-78 

Broad  Area  Maritime  Surveillance  System 
(BAMS) 

AN/SQQ-34  Combat  System 
Brigade  Combat  Team  Modernization 
Program 

Joint  Tactical  Radio  System  (PEO  JTRS) 
Joint  Tactical  Radio  System  Airborne 
Maritime  Fixed  (JTRS  AMF) 

Ground/Air  Task  Oriented  Radar  (GATOR) 
Common  Air  Command  and  Control  System 
(CAC2S) 

Small  Diameter  Bomb,  Incr  II  (SDB  II) 


JMETC  Testing  Success 


Joint  Expeditionary  Force 
Experiment  (JEFX) 

•  Chief  of  Staff  of  the  AF 
directed  series  of 
experiments  that  combines 
LVC  forces  to  create  an 
operationally  representative 
environment  to  assess 
selected  initiatives. 

•  Goal  is  to  rapidly  transition 
enhanced  capability  to  the 
warfighter. 

•  Quarterly  events;  some  Live 
Fly,  some  distributed  LVC 

•  JMETC  Program  support  in 
place  for  two  years 


IMPACT-Cost  Savings! 
•JEFX  Reported  saving  $4.0M  in  FY 
09  using  JMETC  Connectivity  and 
tools 

•Using  JMETC,  JEFX  able  to  now 
complete  3  or  4  distributed  events  per 


year 
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What  are  the  issues? 


•  “Do  I  still  have  the  capability  to  complete  the  mission  following  a 
damaging  event?” 

•  Key  to  Army’s  Mission-Based  Test  and  Evaluation  (MBT&E) 

•  Cannot  be  answered  easily  using  traditional  methods  or  metrics 

•  Not  necessarily  a  single  answer 

•  The  issue  with  using  the  traditional  methods  or  metrics  in  MBT&E: 

•  Traditional  analysis  results  are  qualitative  values  called  loss  of 
function  (LoF). 

•  MBT&E  requires  a  quantitative  understanding  of  a  system’s 
remaining  capability  to  define  an  effect  on  a  mission. 

•  The  correlation  to  a  specific  mission  context  is  not  possible. 


TECHNOLOGY  DRIVEN.  WARFIGHTER  FOCUSED. 
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Objective: 

Create  a  methodology  that  will  quantitatively  map  between  a 
system’s  capabilities  and  a  system’s  components. 

Results: 

•  We  have  developed  the  System  Capabilities  Analytic 
Process  (SCAP). 

•  SCAP  produces  a  map  between  the  system’s  capabilities 
and  the  system’s  components.  These  maps  are  known  as 
the  Functional  Skeleton  (FS). 

•  The  FS  provides  the  information  required  to  determine  the 
remaining  capabilities,  and  therefore  the  course  of  action, 
following  a  damaging  event. 

TECHNOLOGY  DRIVEN.  WARFIGHTER  FOCUSED. 
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Components  that  are  grouped  into 
sub-systems  perform  functions  that 
provide  the  capabilities  to  complete  the 
mission  task. 

SCAP  is  very  similar  to  processes 
used  in  the  consumer-product  industry. 

The  process  reports  metrics  expressed 
in  the  language  of  the  military  user. 

The  focus  of  SCAP  is  a  system’s 
remaining  capability. 


TECHNOLOGY  DRIVEN.  WARFIGHTER  FOCUSED. 
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Sources  of  dysfunction 


Dysfunction  is  defined  as  a  component  that  is  not 
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The  Functional  Skeleton:  A  map 
between  component  and  capability 


Mission  Task 


0 

c 


o 

o 

Q 


CL 


System  Capabilities  System  Functions  Sub-Systems  Components 


Operate  in  Daytime 
Conditions 


Obtain  Location 


T ravel  on  Roads 


Maintain  Internal 
Communications 


Protect  Crew 


Prevent  Catastrophic 
Loss 


Send/Receive 

Short-Range 

Communications 


50  mph 


10  mph 


NOT  Possible 


Generate  Power 


w 


Provide  Fuel 


Accelerate 


Decelerate 


Maintain  Traction 


Support  Weight 


Maintain  Directional 
Control 


Engine  System 


Engine  Block 


Engine  Heads 


Valve  Covers 


Crankshaft 


Camshaft 
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onfljp  How  are  personnel  assessed? 


•  First,  begin  with  the  “battlefield  insult.” 
This  is  the  actual  mechanism  that 
causes  the  injury  /  wounding. 


•  The  injury  is  characterized  both: 

-  in  a  method  to  understand  the  medical 
severity,  and 

-  as  a  detailed  mapping  to  the  ability  to 
perform  certain  functions  post-wounding. 


TECHNOLOGY  DRIVEN.  WARFIGHTER  FOCUSED. 
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What  toolset  assesses  the  crew? 


Operational  Requirement-based  Capability  Assessment  (ORCA) 


Approved  for  Public  Release  -  Distribution  Unlimited 


System 

Capabilities 


System  Functions 


Sub-Systems 


A  high-resolution  „shctline''is  drawn  through  the  affected 
tissues  to  determine  risk  to  life.  This  is  communicated  in 
terms  of  the  Abbreviated  Injury  Scale©  (AIS).* 


None 

Minor 

Moderate 

Serious 

Severe 

0 

1 

2 

3 

4 

The  threshold  of  „3'{serious)  or  greater  is  scored  as  a  medical  casualty. 


Protect  Crew 


Medical  Casualty 

Warfighter  has  experienced  an 
injury  requiring  evacuation  from 
unit  so  that  medical  treatment 
can  be  administered. 


*  Abbreviated  Injury  Scale  ©,  2005,  Updated  2008,  AAAM,  Des  Blaines.  IL,  2008. 
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Operational  Requirement-based  Capability  Assessment  (ORCA) 


Battlefield 

Insult 


Medical 

Assessment 


T 


Medical 

Casualty? 
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m 


Operational  Casualty 


System 

Capabilities 


System  Functions 


Sub-Systems 


Travel  Off-Road 


Operate  Vehicle 


Driver 


Engage  Enemies 


*notional 


Aim  Weapon 


Feed  Weapon 
Ammo 


Fire  Weapon 


Gunner 


0.5 


Incapacitation 


Send/Receive  Short 
Range  Communication 


Communicate  | _ 

via  radio 


Commander 


Operational  casualty  is  defined 
as  a  warfighter  being  unable 
to  perform  a  required  task  or 
function. 


Protect  Crew 


~U 

r 


*notional 


Protect  Crew 
from  Ballistic 


None 

0 


Minor 

1 


Moderate 

2 


Serious 

3 


Severe 

4 


Driver 


Protect  Crew 
from  Fire 


Commander 


Gunner 


Medical  Casualty 

Warfighter  has  experienced  an 
injury  requiring  evacuation  from 
unit  so  that  medical  treatment 
can  be  administered. 
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A  paradigm  shift: 
action-reaction-new  assessment 


In  the  preceding  example,  the  gunner  was  the  only  one  injured. 
After  some  time,  the  Commander  &  Gunner  trade  places*. 


Initial  Incident  (time=0) 

Driver: 


-  AIS:  0 

-  Incapacitation:  0 


Commander: 

-  AIS:  0 

-  Incapacitation:  0 
Gunner: 


-  AIS:  2 

-  Incapacitation  :0.75 


After  Crew  Drill(s) 

Driver: 

-  AIS:  0 

-  Incapacitation:  0 

Commander: 

-  AIS:  2 

-  Incapacitation:  0.1 

Gunner: 

-  AIS:  0 

-  Incapacitation:  0.1 


*assumptions  include  no  deleterious  effects  &  some  loss  of  performance  for  weapon  familiarity  /  zeroing. 


TECHNOLOGY  DRIVEN.  WARFIGHTER  FOCUSED. 

Approved  for  Public  Release  -  Distribution  Unlimited 


Transition  to 
meaningful  results 


Traditional:  mobility  kill 


One  possible  SCAP  metric: 
travel  on  roads 


o 


loss  of  function 


■  can  go  max  speed 
HI  can  go  up  to  30  mph 

IH  can  go  up  to  10  mph 
□  no-go 
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Truck  functional  skeleton 


Because  the  truck  was  damaged, 
it’s  capability  to  travel  on  roads  is 
reduced. 


Conduct 
a  Convoy 


T ravel  on  Roads 


Engage  Enemies 


Protect  Crew 


Prevent  Catastrophic 
Loss 


n 


Generate  Power 


Provide  Fuel 


Accelerate 


Decelerate 


Maintain  Traction 


Support  Weight 


Maintain  Directional 
Control 


LR  Wheel  System  p 

Left  Rear  Rim 


Left  Rear  Tire 


RR  Wheel  System 


LF  Wheel  System 


RF  Wheel  System 


Mission  Task 


System 

Capabilities 


System  Functions  Sub-Systems 


Components 
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System-of -systems  mission 


Two  trucks  are  operating  in  a  convoy 
mission.  By  the  commander'^  intent, 
the  speed  of  the  convoy  is  limited  to 
the  speed  of  the  slowest  vehicle. 


t  -o| — | — 

0  30s 


lOmin 
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t  -M- 

0  30s 


System-of-systems  mission 


Vehicle  not  damaged 


Vehicle  damaged 
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System-of-systems  mission 


t  H— I - 0 - 1 - > 

0  30s  lOmin 


Vehicle  not  damaged 


Vehicle  damaged 
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System-of-systems  mission 


t  -H- 

0  30s 


— I— • - > 

lOmin 


[ 


Vehicle  not  damaged 


Generate  Power 


Maintain  Temp 

I 


Accelerate 

i 


T ravel  on  Roads 


70  mph 


30  mph 


10  mph 


Engage  Enemies 


Protect  Crew 


Prevent  Catastrophic 
Loss 


Commander’s 

Intent 


Vehicle  damaged 


TECHNOLOGY  DRIVEN.  WARFIGHTER  FOCUSED. 

Approved  for  Public  Release  -  Distribution  Unlimited 


•  Further  explore  and  integrate  crew  metrics  and  time-dependent 
degradation 

•  Conduct  SCAP-based  analyses  for  the  MBT&E  pilots  (JLTV,  PIM,  JAGM) 


•  Apply  the  Functional  Skeleton  in  the  System-of-Systems  Survivability 
Simulation  (S4) 

•  Explore  the  utility  of  the  Functional  Skeleton  across  the  Army  enterprise 
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Summary  and  conclusions 


•  ARL/SLAD  has  developed  SCAP  to  quantitatively  map 
between  a  system’s  capabilities  and  a  system’s 
components. 

•  ARL/SLAD  can  use  SCAP  to  generate  quantitative  data 
that  defines  a  system’s  remaining  capability  after  a 
component  is  no  longer  functioning. 

•  Based  on  AEC  feedback,  the  metrics  developed  from 
SCAP  meet  the  requirements  of  MBT&E. 

•  SCAP  has  potential  application  across  the  Army 
enterprise. 
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Existing  Impact 


•  Briefed  at: 

-  2010  March  NDIA  T&E  Conference 


-  2010  October  AORS 

-  2010  August  JLTVLF  IPT 


•  Program  acceptance: 

-  Accepted  by  AEC  as  the  engineering-level  methodology  for  MBT&E 

-  Written  in  the  JLTV  and  PIM  Live-Fire  Strategy 

-  Development  of  Fluman  Availability  Technique  (HAT)* 


•  Publications: 

-  Jan  2010  MBT&E  workshop  first  review  of  SCAP  (ARL-SR-0218) 

-  March  2010  NDIA  T&E  Conference  presentation  of  SCAP  (ARL-SR-0217) 

-  Applying  SCAP  to  the  MBT&E  of  the  JLTV  (ARL-SR-206) 

-  An  Emerging  Methodology:  SCAP  (ARL-TR-5415) 


*  Collaboration  with  HRED 
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IS  ARMY 

RDECOMp  Contact  Information 


William  Landis 

Mechanical  Engineer 
ARL/SLAD 
(410)278-2675 

william.landis1@us.armv.mil 


Richard  Moyers 


Systems  Engineer 


ARL/SLAD 

(410)278-4761 

richard.movers@us.armv.mil 


Kevin  Agan 

Mechanical  Engineer 
ARL/SLAD 
(410)278-4458 

kevin.aaan@us.armv.mil 
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U.S.  Army  Research,  Development  and  Engineering  Command 


Modeling  and  Simulation  for  Mission- 
Based  Test  and  Evaluation  (MBT&E) 


TECHNOLOGY  DRIVEN.  WARFIGHTER  FOCUSED. 


Beth  Ward 

Survivability/Lethality  Analysis  Directorate 
41 0-278-631 5/beth. squier.ward@us.army.mil 


27th  Annual  National  Test  &  Evaluation  Conference  March  14-17,  201 1 


The  purpose  of  this  presentation  is  to  provide 
background  on  MBT&E,  supporting  tools,  and 
modeling  and  simulation  (M&S)  applications. 


Bottom  line  up  front:  M&S  used  in  testing  need  to 
expand  the  linkages  between  materiel  attributes 
and  operational  capabilities  for  MBT&E. 


TECHNOLOGY  DRIVEN.  WARFIGHTER  FOCUSED. 
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_  US  ARMY 

0  RDECOMJk  Outline 


•  Why  and  what  is  MBT&E? 

•  Approaches  to  organizing  an  effective 
M&S  program  for  MBT&E 

•  M&S  issues 

•  What  are  we  doing  to  solve  the  issues? 

•  Summary 

•  Points  of  contact 


TECHNOLOGY  DRIVEN.  WARFIGHTER  FOCUSED. 
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A£/EC  Why  was  MBT&E  developed? 

•  Drive  operational  mission  context  into  all  test  and 
evaluation  (T&E). 

•  Develop  a  T&E  methodology  that  fully  addresses  recent 
acquisition  initiatives. 

•  Provide  “feedback”  directly  to  the  joint  capabilities 
integration  and  development  system  (JCIDS)  in  terms  of 
the  war  fighter’s  mission. 

•  Enable  robust  and  systematic  system-of-systems  T&E. 

Director, ;  Operational  Test  and_  Evaluation  -  “The  evaluation  of 
operational  effectiveness  [  and  system  performance]  is  linked  to 

mission  accomplishment.  ^ 


1.  Memorandum,  OSD  DOT&E,  subject:  Reporting  of  Operational  Test  and  Evaluation  Results,  6  Jan  10. 


Army  Proven 

Bottle  Ready 


Courtesy  of  Chris  Wilcox,  Army  Evaluation  Center,  ATEC 
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What  is  MBT&E? 


Mission-Based  Test  and  Evaluation 

is  a  methodology  that  focuses  T&E  on  the 
capabilities  provided  to  the  warfighter.  It  provides  a 

framework  an6  procedure  to: 

-  link  materiel  system  attributes  to  the  operational 
capabilities; 

-  examine  the  SoS  required  to  enable  the  operational 
capability;  and 

-  examine  synergistic  use  of  all  available  data  sources. 


Army  Proven 

Bottle  Ready 


Courtesy  of  Chris  Wilcox,  Army  Evaluation  Center,  ATEC 
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0  roecom  >  Approaches  to  organize  an 

effective  M&S  program 


•  Tools  for  test  and  evaluation  planning 

•  The  test  and  evaluation  support  tool  and  example 
repository  (TESTER) 

•  Model-based  systems  engineering  with  Vitech  CORE 


•  Models  and  simulations  to  augment  costs  of  testing 

•  OneSAF  (semi-automated  forces) 

•  Infantry  Warrior  Simulation  (IWARS) 

•  Combined  Arms  Analysis  Tool  for  the  21 st  Century  (COMBAT  XXI) 

•  System  of  Systems  Survivability  Simulation  (S4) 

•CORE 


Critical  to  an  effective  M&S  program  is  to  understand  model 
purpose,  requirements,  timelines,  and  limitations. 
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TESTER:  Online  MBT&E 


TIJTIR#r 


Users 

•  Army  Evaluation  Center 
(AEC)  Evaluators 

•  AEC  System  Team 
(AST)  Members 

■  Operational  Test 
Command  (OTC) 

■  Developmental  Test 
Command  (DTC) 

■  Analysts 

■  Modeling  & 
Simulation 
Representatives 

•  Other  Stakeholders 

■  Program  Manager 

■  Training  &  Doctrine 
Command 
(TRADOC) 

■  Test  Centers 


Access 
System  via 
CAC  Login 


List  of  Current  Systems  is  provided 
by  an  Army  Online  Database 


TESTER  Process  and  Functions 


Collect  Source 
Documentation 


Develop 

Measures 


Identify  Issues 
&  Standards 


•  Key 

Performance 

Parameters 

•  Key  System 
Attributes 

•  Critical 
Operational 
Issues 

•  Etc. 


Define  Missions  Identify  Component! 


&  Tasks 


1 


&  Functions 


'  -fV* 


■Fire 
•  Protect 
■Maneuve 
•Sense 
■Etc. 


Link  Components  &  Functions 


to  Missions  &  Tasks 


Capture  Design  of  Experiments 


— 

Factor 

Factor  Level 

Data  Source 

P/S 

Control  Technique 

Flat 

LUT 

P 

Field  Constant 

Terrain 

Rolling 

OneSAF 

P 

Tactically  Varied 

LUT 

P 

Uncontrolled 

Light  Level 

Full  Sun 

OneSAF 

s 

Field  Constant 

Night 

LUT 

p 

Field  Constant 

Weather 

Rain 

OneSAF 

p 

Systematically  Varied 

Dust 

OneSAF 

p 

Random  Assignment 

Identify  Data 
Requirements 


Identify  Data 
Sources 


What  data  will 
need  to  be 
collected  from 
test  to  answer 
measures. 


Develop  Reports 

Data  Source  Matrix 

ST  -  KTSB  -S5S- m  -  iSstt'-TT-’S 


Army  Proven 

Baffle  Ready 


TESTER  will  streamline  MBT&E  System  Evaluations  and  facilitate  collaboration 
among  distributed  System  Teams  and  other  stakeholders. 


Reports  can  be 

generated  to: 

•  Enable 
System 
Evaluations 

•  Assist  in  Test 
Planning 

•  Facilitate 
Design  of 
Experiments 
planning  and 
execution 

•  Ensure  all 
needed  data 
is  collected  for 
system 
evaluation 


Courtesy  of  Jamie  Pilar,  Army  Evaluation  Center,  ATEC 
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Systems  Engineering  with  CORE 


Utilizing  a  layered  approach  to  progressively  clarify  and  elaborate  all  four  domains 
concurrently  ensures  consistency  and  completeness. 
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_  US  ARMY 

0  rdecowQ  M&S  issues 


The  MBT&E  strategy  presents  several  issues  in  the  application  of 
modeling  and  simulation  (M&S)  to  augment  testing  limitations  and 
associated  costs: 


•  The  vulnerability/lethality  (V/L)  data  and  usage  of  that  data  in 
traditional  M&S  does  not  meet  the  requirements  for  MBT&E. 

•  Historically,  V/L  data  were  generated  by  multiplying  an  average  combat 
utility  value  to  a  loss-of-fu notion  (LoF)  probability  (i.e.,  how  well  the  system 
can  perform  its  mobility  [M]  or  firepower  [F]  functions). 

•  In  Army  M&S,  the  LoF  values  are  then  applied  to  all  possible  combat 
scenarios*. 

•  MBT&E  aligns  system  components  and  functions  to  a  specified 
tactical  mission  at  a  higher  resolution  than  M/F  LoF. 

•  The  approach  then  evaluates  system  capability  requirements  of  a  mission 
in  addition  to  technical  performance  parameters. 


•  M&S  used  in  testing  need  to  expand  the  linkages  between 
materiel  attributes  and  operational  capabilities. 

*  Deitz,  Paul  H.,  and  Starks,  Michael  W., 

“The  Generation,  Use,  and  Misuse  of  “PKs”  in  Vulnerability/Lethality  Analyses”,  TECHNOLOGY  DRIVEN.  WARFIGHTER  FOCUSED. 

U.S.  Army  Research  Laboratory,  APG,  MD.,  ARL-TR-1640,  MAR  1998. 


RDECOM 


MBT&E  metrics  example: 

materiel  system  attributes 


System  Capabilities  Assessment  Process  (SCAP) 
Functional  Skeletons 

Cateaorv 

Svstem  Caoabilitv 

SC  bin 

Move 

Travel  on  primary  roads 

can  go  max  speed 

primary  up  to  50  mph 

primary  up  to  30  mph 

primary  up  to  10  mph 

no-go 

Travel  off  roads 

can  go  max  speed 

primary  up  to  50  mph 

primary  up  to  30  mph 

primary  up  to  10  mph 

no-go 

Travel  cross-country 

up  to  28  mph 

up  to  18  mph 

up  to  5  mph 

Emplace 

Pivot  steer 

360°  /  10  sec 

no-go 

Start  engine 

fully  capable 

no-go 

Shoot 

Fire  standard  munition 

4  rounds  /  min 

1  round  /  min 

NOT  Possible 

Fire  self-defense  gun 

Fully  Capable 

no-go 

Aim  main  gun  -  direct  fires 

automatic  lay 

manual  lay 

no-go 

Aim  main  gun  -  indirect  fires 

automatic  lay 

manual  lay 

Survive 

Protect  Crew 

protect  crew  from  ballistic 

protect  crew  from  CBRNE 

protect  crew  from  rollover 

Prevent  catastrophic  loss 

protect  all  energetic 

protect  Munitions 

protect  Propellant 

protect  Fuel 

no-go 

Protect  from  NBC 

Control  fires  (AFES' 

) 

fully  capable 

no-go 

Protect  from  gun  backblast  /  byproducts 

Maintain  internal  enviromental  conditions 

Rapid  egress 

open  all  access 

bin  2 

no-go 

Prevent  visible  detection 

Prevent  thermal  detection 

Prevent  signals  detection 

Observe 

Operate  during  day 

fully  capable 

no-go 

Operate  during  night 

fully  capable 

no-go 

Operate  obscured 

Define  bins  with  TRADOC 

Identify  location 

GPS 

vehicle  motion 

no-go 

Provide  navigation 

IFF 

Communicate 

Communicate  short  range 

Fully  capable 
data  only 
analog/voice 
no-go 

Communicate  long  range 
SATCOM  all  crew 

Communicate  intra-system 

fully  capable 

Communicate  inter-system 
Communicate  to  dismount 
Unique  attribute 

Ammunition  reload 
Haul  vehicle 

Provide  power  from  slaved  vehicle 


TECHNOLOGY  DRIVEN.  WARFIGHTER  FOCUSED. 
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ARL  has  developed  the  task-system  capability  matrix  and 
functional  skeletons  for  the  High  Mobility  Multi  Wheeled 
Vehicle  (HMMWV)  and  the  Joint  Light  Tactical  Vehicle 
(JLTV). 

The  challenge  is  to  determine  how  Army  M&S  can  use  these 
new  metrics  to  benefit  the  evaluator. 


TECHNOLOGY  DRIVEN.  WARFIGHTER  FOCUSED. 
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rdecom  jm  Model  resolution  and  metrics 


Use/Study 


Resolution 


Study  Timeline* 


Considerations 


AWARS 

Analysis  of 
Alternatives 
(AoA) 

Division  and 
Brigade:  Entity 
Level 

Outside  MBT&E 
requirement 

Aggregate  metrics 
built  from  high 
resolution  data 

OneSAF 

AoA, 

Training, 

T&E 

Brigade  and 
Below:  Item 

Level 

Years 

Formal  process  for 
requirements  outside 
ATEC  control  but  used  in  OT 

IWARS/ 

COMBATXX1/ 

S4 

AoA, 

SoSA, 

Many  on  many 

Brigade  and 
Below:  Item 

Level 

Months 

AMSAA  M&S 
cell  and  studies 
could  be  leveraged 

Ground  Wars 

AoA 

Few  on  few 

Platoon:  Item 

Level 

Weeks 

Earlier  efforts  can  be 
leveraged  to  provide 
limited  capability 

RTCA 

Operational 

Assessment 

Platoon:  Item 

Level 

Months 

New  metrics  in  M&S 
at  ATEC 

CORE 

Engineering  and 
Requirements 

Platform:  Item 

Level 

Weeks 

System  characterization 
repository  linked  to 
requirements 

-  MBT&E  metrics  must  replace  loss-of-function  data 

-  Decision  tables  must  be  developed  to  ‘act’  on  system  attributes  (remaining  capability) 


.  .  .  .  .  .  ,  t.  .  TECHNOLOGY  DRIVEN.  WARFIGHTER  FOCUSED. 

Timeline  includes  model  development,  data  generation  and  analysis 
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(TJ  RDECOMJ >  What  are  we  doing  to 

solve  the  issues? 


•  SLAD  is  collaborating  with  AMSAA ,  ATEC  and  TRADOC  to  develop  an 
M&S  methodology  for  MBT&E. 

•  Significant  actions 

1)  Establish  a  Language  and  Definition  Working  Group.  Purpose  is 
to  develop  a  lexicon  for  the  terms/definitions  used.  An  additional 
purpose  would  be  to  develop  and  coordinate  a  common  framework  that 
will  support  TRADOC,  AMSAA,  ARL  and  ATEC. 

2)  Develop  a  category/attribute  template. 

Can  be  done  in  conjunction  with  the  language  and  definition  working 
group.  Purpose  is  to  develop  a  universal  set  of  attributes  (and  attribute 
levels)  that  sets  the  stage  for  rational  of  desired  capabilities. 

3)  Establish  a  Scenario  Utility  Working  Group. 

Purpose  is  to:  (a)  learn  what  TRADOC  does  and  how  they  do  it  when 
they  develop  scenarios;  and  (b)  provide  feedback  from  RD  and  T&E 
communities  as  to  what  we  are  looking  for  and  how  TRADOC's 
scenarios  can  support  what  we  need. 

SLAD,  in  collaboration  with  AMSAA,  will  propose  how  MBTE 
metrics  could  be  used  by  TRADOC  models. 

TECHNOLOGY  DRIVEN.  WARFIGHTER  FOCUSED. 
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(TJ  RDECOMJ >  What  are  we  doing  to 

solve  the  issues? 


SLAD  met  with  AMSAA  SMEs  to  discuss  ideas  to  develop  a  M&S  test  bed  for 
MBT&E. 

One  approach  to  a  M&S  development  could  begin  with  a  small  unit  simulation 
for  high  resolution  data  then  incrementally  progress  to  a  larger  simulation 
for  lower  resolution  data  (i.e.,  aggregated  MBT&E  metrics). 

The  expected  results  from  the  experimentation  would  include 

•  methods  to  input  MBT&E  metrics, 

•  algorithms  for  data  usage, 

•  method  to  aggregate  MBT&E  metrics  for  higher  level  M&S, 

•  analysis  techniques,  and 

•  recommended  practices. 


TECHNOLOGY  DRIVEN.  WARFIGHTER  FOCUSED. 
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US  ARMY 

0  rdecom 'j)  Considerations 


•  MBT&E  encompasses  more  than  LFT  in  support  of  Army 
acquisition. 

•  ATEC  must  render  evaluations  based  upon  system  use  to 
accomplish  combat  missions  (Joint  context) 

•  Technical  leadership  is  looking  for  higher  resolution 
modeling  to  support  evaluations  with  goals  to  include 

•  improve  understanding  of  data  metrics 

•  incorporate  consistent  data  development  methods  and 
usage  across  varying  resolutions 

Desired  end-state  is  a  level  of  consistency 
in  the  metrics  for  Army  acquisition. 


TECHNOLOGY  DRIVEN.  WARFIGHTER  FOCUSED. 


15 


Critical  to  an  effective  M&S  program  is  to  understand  model 
purpose,  requirements,  timelines,  and  limitations. 

The  MBT&E  strategy  presents  several  challenges  in  the  application 
of  M&S,  test  planning/execution,  and  the  analysis  of  data  for  system 
evaluation. 

AEC  development  of  TESTER  will  streamline  MBT&E  system 
evaluations  and  facilitate  collaboration  among  distributed  System 
Teams  and  other  stakeholders. 

M&S  used  in  testing  need  to  expand  the  linkages  between  materiel 
attributes  and  operational  capabilities  for  MBT&E. 

SLAD  is  collaborating  with  multiple  agencies  to  help  develop  the 
methodology  to  make  those  linkages  possible  in  M&S. 

TECHNOLOGY  DRIVEN.  WARFIGHTER  FOCUSED. 
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Beth  Ward, 

Survivability/Lethality  Analysis  Directorate 
41 0-278-631 5/beth.  squier.ward@us.army.mil 

Chris  Wilcox,  MBT&E 

Army  Evaluation  Center 

41 0-306-21 93/Chris. Wilcox1@us.army.mil 

Jamie  Pilar,  TESTER 

Army  Evaluation  Center 

41 0-306-21 93/Chris. Wilcox1@us.army.mil 

Ken  Helton,  CORE 

Vitech  Corporation 
M:  540.239.1424/ 
community.vitechcorp.com 


TECHNOLOGY  DRIVEN.  WARFIGHTER  FOCUSED. 
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27U  Annual  National  Test  and  Evaluation  Conference 


Commercial  Ilf  Iff  visos 
MT&E  Program  Manager 


16  March  2011 


Contract  Industry 

Fleet  Training 

Foreign  Military  Sales  Support 
RDT&E 

NAVAIRSYSCOM  PMA-207  CAS  concentrates  of  OP4  (Opposing  Force  Training) 
for  the  fleets.  EW  training,  target  towing,  JTACT,  and  Airborne  tanking 

NAWCAD  CAS  supports  FMS  and  RDT&E  tasking  for  DoD. 

F-18  NRL  ARROW  Program 

E-2C/D  RAAF 

Mode  5  Canadian  Forces 

MIT  (Advance  Radar)  RAF 


IZK2  Ur 


6  aircraft  split  between  NBVC  Ft  Mugu,  CA  and  Newport  News,  VA 


I 


Cartersville,  6A. 

Utilized  primarily  for  Range  Clearing 


6-1  Gnlfstream 


13  Aircraft-  Located  at  North  island,  San  Diego,  CA  island  Newport  News,  lM 


61  AIRCRAFT  9  DIFFERENT  TYPES 


LEAR  RDT&E  AIRCRAFT 


LEAR  GENERIC  EQUIPMENT  RACK 


LEAR  BUZZER  AIRCRAFT 


LEAR  NOSE  ANTENNA  MOUNT 


LEAR  TAIL  ANTENNA  MOUNT 


LEAR  RDT&E  MODIFICATIONS 


•  Mission  Power  (9KVA,  60  Hz,  28VDC) 

•  Dual  ARC-164  Cockpit  UHF  Radios 

•  Military  ARN-1 1 8  TACAN  System 

•  TSPI  Data  (GPS  position,  pitch,  roll,  heading, 
altitude) 

•  GPS  Antenna  Splitter  (1x4) 

•  Operators  ARC-182  VHF/UHF  radio 

•  D  Band  Antennas  (5  ea) 

•  C  Band  Antennas  (3  ea) 

•  Nose  and  Tail  Antenna  Cabling  and  Locations 

•  Tracking  Beacon  Antenna 

•  Port  and  Starboard  Wing  RF  Cabling 

•  Generic  19”  Equipment  Racks 

•  Divan  Rack  Test  Equipment 


LEAR  RDT&E  SYSTEMS 


Military  IFF  (APX-72  and  APX-123) 
IFF  Buzzer  (nose  and  tail) 

B  Band  Buzzer  (nose  and  tail) 

GPS  Buzzer  (nose  and  tail) 

DRFM  (nose  and  tail) 


•  A  central  point  for  air  assets 

•  Available  To  Government  Agencies  And  Civilian  Contractors  Supporting  Government 
Projects 

•  Accessible  To  Allied  Forces 

•  OPSEC  Capable 

•  Extensive  Electronic  Warfare  Experience  In  Military  Training,  and  supporting  RDT&E  projects 

•  29  Years  Experience  Working  With  The  FAA,  Contractors,  and  DOD 

•  FAA  Certification 

•  Airworthiness  oversight. 

•  Engineering  Oversight  Office 

•  Worldwide  Support  Capabilities 

•  Aircraft  Airframe  And  Avionics  Modification  To  Meet  Project  specifications 

•  In-House  Or  Can  Access:  EA  /  ECM  /  Threat  SIM  /  Chaff  Pods,  Tow  Targets,  Prototype  EW 
Systems 


16  Marti  2011 


Mission  Based  T&E 

Progress 


Christopher  Wilcox 

Deputy/Technical  Director 
Fires  Evaluation  Directorate,  US  AEC 

Army  Proven  15  Mar  11 

Baffle  Ready 


Purpose  and  Agenda 


•  Purpose:  To  review  the  status  of  the  MBT&E  methodology  in  the 
following  areas: 

-  Implementation, 

-  Lessons  Learned,  and 

-  Current  Development  Focus  Areas. 


•  Agenda 

-  Background  (Why  and  What) 

-  Implementation  (How) 

-  Lessons  Learned  (Items  to  Sustain  and  Improve) 

-  Current  Development  Focus  Areas 

-  Conclusions 

Army  Proven 

Battle  Ready 
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A&EC  Why?  -  Acquisition  Initiatives 

Common  Focus  on  Mission  Capability 

- DoD - 

POD  5000.1  -  “The  primary  objective  of  Defense  acquisition  is  to  acquire  quality 
products  that  satisfy  user  needs  with  measurable  improvements  to  mission 
capability...”1 

- JCS - 

Joint  Capabilities  Integration  and  Development  System  -  The  primary 
objective  of  the  JCIDS  process  is  to  ensure  the  capabilities  required  by 
the  joint  warfighter  are  identified  ...  in  order  to  successfully  execute 
the  missions  assigned.”2 

-  DOT&E - 

Director,  Operational  Test  and  Evaluation  -  “The  evaluation  of 

operational  effectiveness  is  linked  to  mission  accomplishment.”3 


Goal:  T&E  Focused  on  Mission  Capability 


Army  Proven 

Battle  Ready 


1.  Office  of  the  Under  Secretary  of  Defense,  Acquisition,  Technology  and  Logistics,  Department  of  Defense  Directive  Number  5000.1,  12  May  2003. 

2.  Office  of  the  Joint  Chiefs  of  Staff,  Chairman  of  the  Joint  Chiefs  of  Staff  Instruction  31 70.01  G,  1  Mar  09. 

3.  Memorandum,  OSD  DOT&E,  subject:  Reporting  of  Operational  Test  and  Evaluation  Results,  6  Jan  10. 
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What?  ■  Framework  Building  Block 

Capability1  -  The  ability  to  achieve  a  desired  effect  [or  result, 
outcome,  or  consequence  of  a  task2] . . . 


-  under  specified  standards  and  conditions 

-  through  a  combination  of  means  and  ways 


1.  CJCSI  31 70.01  F,  May  2007 

Army  Proven  2.  Taken  from  JP  1-02,  Mar  2007,  definition  of  effect. 

Battle  Ready 
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What?  -  MBT&E  Framework 


MISSION  AND  SYSTEM 


Mission  Capabilities 


EVALUATED  BY 


(Higher  Commander  ,3  mission  and  tasks) 


Task 


Desired  Effect 


MISSION  PLANNING 


\ 


ENABLES 


SoS  Task  Capabilities 

(Mission  and  tasks  of  unit  employing  the  system)  V 


Task 


Desired  Effect 


SYSTEMS  ENGINEERING 


Measures 

Of 

Effectiveness 


ENABLES 


Army  Proven 

Baffle  Ready 


Measures 

Of 

Performance 


TESTED  BY 

Contractor 

Testing 

Developmental 

Testing 

Live  Fire 
Testing 

Operational 

Testing 

Models  & 
Simulations 

Demonstrated 

Certifications 
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1M 


What?  -  Putting  it  all  together 

Link  Measures  to  Data  Sources 


How?  -  Strategy  Development 


The  T&E  Strategy... 

•  Initial  strategy  development  using 
MBT&E  derived  template; 

•  Links  the  attributes  of  the  system  to 
mission  context;  and 

•  Addresses  Critical  Operational 
Issues,  Key  Performance 
Parameters  in  the  mission  context. 


Mission  context  driven  from 
evaluation  strategy  through  DT 
and  OT. 


Context  for 


~aieo 


DATA  SOURCES 


_ M'.-  V _ 


DOE 


~^EC 


T&E  CONCEPT 


"47ec 


CAPABILITIES 


"47er 

a 


SYSTEM 


~aJeo 


TASKS 


~AJec 


MISSION 


Project:  Husky  MK  I 


Route  Clearance  Squad  Deliberate  Route  Clearance  Formation 


Unit:  CENTCOM  Engineer  Route  Clearance  Squads  in  OEF 

Operational  Mode  Summary:  Improved  &  Unimproved  Routes 

ART  1 .0  Movement  and  Maneuver 
EFFECTIVENESS  ART  1.6.1. 1  Conduct  Breaching  Operations 
ART  1 .6.1.2  Conduct  Clearing  Operations 
ART  4.0  Sustainment 
ART  4.1. 1.1  PMCS 

SUITABILITY  ART  4.1. 1.2  Recover  &  Evacuate  Equipment 

ART  4.1 .1 .6  Repair  Equipment 
ART  4. 1.3.9  Provide  Repair  Parts 
ART  6.0  Protection 

ART  6. 7.1.1  Protect  Individuals  &  Systems 
ART  6. 7. 1.4  Employ  Protective  Equipment 


SURVIVABILITY 


Army  Proven 

Battle  Ready  1 


Army  Evaluation  Center 


Army  Proven 

Bottle  Ready 
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How?  -  Use  of  Authoritative  Task  List 


MBT&E  Process:  1.  Develop  mission  tasks.  2.  Link  to  ATL 


Army  Universal  Task  List 


T&E  Plan 


ART  1.4.1  CONDUCT  LETHAL  DIRECT  FIRE  AGAINST  A  SURFACE 
TARGET 

1-54  ~npFi  tuomy  wpupraz  aod  1  pticiomcJ.  fbod5:jtiaii  ilJ  zi:b-w\  wrii  Jr*“  iir& 
k  i nraw.  TlifrfH  dine  £tk  mi-  "w  ir  :ci.  or  —Tzrm-  vinE  v-socii  iTiS  3-W- 

(UEAU£) 


,V|1. 

SAAfr 

Jtanv 

ai 

YeslMo 

Cl-ecl'rw  innTttJEdtaKo^sHno  mil  nteswi. 

02 

Y«.Mo 

□tadfl  re  :H3:r  was.  conducro  ce^  eila:-  shed  rues  of  enaage'nert. 

03 

YeiT^o 

Un :  l:-*:  co^col  a-hc^"  t  erqase  lyat: 

•- 

04 

Tire 

To*  nef  cd^x  ? k  Mtazfc  or  3  reel 'r*  tne  a^cr  cefcrlrs  and  rterrV  '%  la^er.  9l* 

OE 

Tine 

To-  sjppte:  “arocSs. 

3E 

Percent 

O'  [r:(ai' t/  of  SrLppneserg  a  target 

07 

PbldI 

O'prcta&lty  ofa  fit 

as 

Percent 

Of  prot-aa-i  ly  of  a  UO  ohm  a  Itt  9- 

as 

Percent 

O'  m  ssto^:  arti  rrec  1:  acheve  target  anas*. 

10 

O'  a-ra  able  dtadfre  weaocr  ^/s-tera  engaging  ilrestl  Tit  -.argils  9k 

11 

Percent 

O'  dfc-eci  rre  teigefc,  rot  engaged 

12 

Fmrt 

O'  cne^  s-er^or-anc*  due  to  ettia  dlTcf  rre  attach. 

13 

Percent 

O' edtal  dnectlT  alaocs-tra:  TS^JIIn  :olatera  oanage. 

14 

PemerTt 

O'  lethal  o  Feet  Inc  attacks.  Tat  tsJI  In  ^endy  a  nettral  casafce:. 

IE 

Mirnbef 

Offers  drecflT  attacSsra  TsJIln  colaisra  danage. 

15 

WirtBf 

O'  eYal  ■:  red  It  attach  Tat  Tetfl  In  trendy  cr  neutral  :as>_at:*:. 

EFFECTIVENESS 

ART  1.4.1:  DIRECT  LETHAL  FIRES 
End  State:  Target  is  destroyed 


>  MOE:  %  Correct  Weapon  Settings 
MOE:  Time  to  Attack 
MOE:  Probability  of  kill 
MOE:  %  Targets  Engaged 
MOE:  %  Collateral  Damage 


Army  Proven 

Boffle  Ready 
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How?  -  Planning 


Alec 

The  T&E  Plan... 


•  Focuses  on  Soldier  missions 
and  tasks; 

•  Links  the  attributes  of  the 
system  to  mission  context;  and 

•  Addresses  Critical  Operational 
Issues,  Key  Performance 
Parameters  in  the  mission 
context. 

r . ' . " . ' . . . .  " . . . . . . . " . ' . ' . 

Mission  and  task  capabilities  are 
the  highest  level  of  the  T&E 
dendritic. 

" _ _ _ w 


Mobile  Tower  System  Evaluation  Plan 

CHAPTER  3.  EVALUATION  DETAILS . 

3.1  EFFECTIVENESS . 

3.1.1  MBA  1  Night  Vision  Device  Compatibility . 

3.1.2  Mission  Task  2  Install  the  MOTS . 

3. 1.2.2  Task  2.2  Setup  and  Tear  Down . 

3. 1.2.3  Task  2.3.  Conduct  Minimum  Initial  Operations . 

3.1 .3  Task  3.  Conduct  Tower  Operations . 

3.1 .3.1  Mission  Task  3.1  Retrieve  Recorded  Communications . 

3. 1.3.2  Task  3.2  Operate  Airfield  Lighting  System . 

3. 1.3.3  Task  3.3.  Obtain  Airspace  Information  and  Send  Messages  Via  TAIS . 

3.1 .3.4  Task  3.4  Control  Aircraft,  Vehicles,  and  Personnel  by  ATC  Light  Gun 

Signals . 

3. 1.3. 5  Mission  Task  3.5  Communicate  Using  RF  and  Landline . 

3.1 .3.6  Task  3.6  Provide  Local  Wind  Speed/Direction/Altimeter  Setting . 

3.2  SUITABILITY . 

3.2.1  MEA  2  Training  and  Training  Devices . . 

3.2.2  MEA  3  Reliability,  Availability,  and  Maintainability . 

3.2.3  MEA  4  Integrated  Logistics  Support  (ILS) . 

3.2.4  MEA  6  System  Safety . 

3.2.5  Task  1  Transport  the  MOTS . 

3.2.6  Task  4  Maintain  the  MOTS . 

3.2.7  Mission  Task  5  Train . 

3.3  SURVIVABILITY . 

3.3.1  MEA  5  System  Survivability . 

3.3. 1.1  MEA  5.1  Electromagnetic  Environmental  Effects . 

3. 3. 1.2  MEA  5.2  Information  Security . 

3. 3. 1.3  MEA  5.3  Chemical,  Biological,  Radiological,  and  Nuclear  Effects 


Army  Proven 

Bottle  Ready 


MEA:  Mission  Enabling  Attribute.  MOTS:  Mobile  Tower  System 
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How?  -  Reporting 


•  OTA  Evaluation  Report 


•  Conclusions  focused  on  Soldier 
tasks  and  how  the  system 
supports  the  mission. 

•  COIs,  Criteria  and  KPPs 
addressed,  but  conclusions  are 
put  in  the  context  of  the  Soldier’s 
mission  and  tasks. 


Route  Clearance  and 
Proofing  System 


CHAPTER  2,  CONCLUSIONS . 2-1 

2.1  EVALUATION  OF  OPERATIONAL  CAPABILITY . 2-1 

2.1.1  Effectiveness . 2-1 

2. 1 . 1 . 1  ART  1 .6. 1 . 1 .  Conduct  Breaching  Operations . 2-1 

- 2. 1.1.2  ART  1.6. 1.2.  Conduct  Clearing  Operations . 2-2 

2.1.2  Suitability . 2-2 

2. 1.2.1  ART  4. 1.1.1.  Perform  PMCS . 2-2 

2. 1.2.2  ART  4. 1.1. 2.  Recover  and  Evacuate  Disabled  Equipment . 2-3 

2. 1.2.3  ART  4. 1.1. 6.  Repair  Equipment . 2-3 

2. 1.2.4  ART  4. 1.2.2.  Conduct  Terminal  Operations . 2-4 

2.1. 2.5  ART  4.1. 2.3.  Conduct  Mode  Operations . 2-5 

2. 1.2.6  ART  4. 1.3.9.  Provide  Repair  Parts  (Class  IX) . 2-5 

2.1.3  Survivability . 2-5 

2. 1.3.1  ART  6.7. 1.1.  Protect  Individuals  and  Systems. . 2-5 

2. 1.3.2  ART  6.7. 1.4.  Employ  Protective  Equipment . 2-5 


All  T&E  results  are  related  to 
the  mission. 


Army  Proven 

Baffle  Ready 


2.1 .1.1  ART  1 .6.1.1  Conduct  Breaching  Operations 

-  End  State:  “creation  of  lanes  through  or  over  an  obstacle  to  allow  an 
attacking  force  to  pass.” 

-  Result:  “The  SYSTEM  supports  this  task  by  detecting  the  threat 
obstacle,  marking  the  threats  (for  interrogation)  and  towing  the 
clearing  set  to  „proofthe  lane.  The  SYSTEM  ...  is  a  significant 
improvement  over  dismounted  IED  detection,  marking  and  proofing.” 
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r^EC  items  to  Sustain  ■  Planning 

•  MBT&E  strategies  being  developed. 

-  Linking  all  T&E  requirements  to  missions  /  tasks. 

-  Leveling  of  expectations  in  T&E  IPT. 

•  Mission  context  enhancing  T&E  design. 

-  Mission  context  (desired  results,  conditions,  standards)  leads  to  integrated  T&E. 

-  Evaluation  measure  design  focused  on  operational  capability. 

-  DT  designed  using  operational  techniques  and  procedures. 

•  SoS  description  aligned  with  PM’s  Work  Breakdown  Structure. 

-  Facilitates  sharing  of  T&E  data  during  contractor  testing. 

-  Aligns  Warfighter  tasks  with  contractor  requirements. 

Mission  context  and  SoS  description  - 
keys  to  integrated  T&E  strategy 

Army  Proven 

Bottle  Ready 
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r$EC  items  to  Sustain  ■  Reporting 

•  Mission  Task  to  System  Attribute  Linkages. 

-  Understanding  how  system  technical  performance  impacted  desired 
capabilities. 

-  “Accumulated”  evaluation  of  effectiveness,  suitability  and  survivability. 

•  Conclusions  more  than  a  restatement  of  test  results. 

-  MBT&E  Capabilities  =  task  +  desired  result. 

-  Conclusions  telling  “what  the  data  means”  in  terms  of  capabilities. 


Answering  the  “so  what”  question  in  the  Warfighter’s  terms 


Army  Proven 

Bottle  Ready 
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Items  Being  Improved  ■  Planning 

•  Linkages  between  tasks  and  system  attributes  are  being 
developed. 

-  Impact:  Additional  time  to  develop  and  coordinate  linkages. 

-  Mitigation:  T&E  IPT  developing  during  project  execution. 

-  Path  ahead:  Develop  linkages  as  capabilities  based  analysis  is  being  conducted. 

•  Reference  missions  and  tasks  are  being  developed. 

-  Impact:  Additional  time  to  develop,  coordinate  and  “validate”  reference  missions. 

-  Mitigation:  Direct  coordination  with  TRADOC  School  Houses. 

-  Path  ahead:  Develop  set  of  reference  mission/tasks  per  Warfighting  Function. 


Army  Proven 

Bottle  Ready 
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Items  Being  Improved  ■  Reporting 

•  Mission/task  standards  (threshold/objective  requirements)  are 
being  developed. 

-  Impact:  Qualitative  results  solely  based  on  military  judgment. 

-  Mitigation:  T&E  IPT  developing  “expected”  mission/task  performance. 

-  Path  Ahead:  Develop  task,  conditions  and  standards  in  requirements. 

•  Roll-up  of  system  and  operational  performance  into  overall 
assessment  of  ESS  is  being  developed. 

-  Impact:  ESS  still  based  on  met/not  met  technical  requirements.  Impact  of 
sustainability/survivability  on  effectiveness  not  determined. 

-  Mitigation:  Providing  capabilities  and  limitations  as  rationale  for  ESS  assessment. 
Continue  to  use  links  to  COIs  and  KPPs  in  parallel. 

-  Path  Ahead:  Align  Critical  Operational  Issues/Criteria  with  mission  and  tasks. 

Army  Proven 

Bottle  Ready 
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Current  MBT&E  Development  Focus 

•  Developing  better  understanding  of  the  mission  context. 

-  How  will  the  Warfighter  execute  the  mission? 

-  What  is  needed  to  execute  the  mission? 

-  Under  what  operational  conditions  are  the  capabilities  needed? 

•  Incorporating  mission  analysis  into  the  requirements  development  process. 

-  What  are  the  key  Warfighter  capabilities  (task  +  desired  result)  needed  for  the  mission? 

-  How  do  you  know  that  the  capabilities  are  supporting  mission  accomplishment? 

-  How  do  the  attributers,  KPPs,  and  COIs  support  assessment  of  capabilities? 

•  Incorporating  relationship  between  Systems  Engineering  and  war  fighter  Task. 

-  How  do  the  SoS  components  support  the  tasks? 

-  What  level  of  technical  performance  is  necessary  to  support  task  accomplishment? 

Collaboration  between  Combat  Developer,  Materiel  Developer  and 

Independent  T&E. 

Army  Proven 

Baffle  Ready 
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Conclusions 


•  Implementation  of  MBT&E  is  showing: 

-  Mission  and  task  capabilities  are  highest  level  (focus)  of  T&E  strategy  =  results  related  to  mission. 

-  Providing  conclusions  in  Warfighter’s  terms. 

-  Mission  context  driven  into  DT  and  OT  conduct  =  integrated  T&E  programs. 


•  Items  to  Sustain: 

-  Use  of  ATLs,  and  especially  the  AUTL,  as  source  of  evaluation  metrics. 

-  SoS  description  aligned  with  PM’s  Work  Breakdown  Structure. 

-  Use  of  mission  context  and  SoS  description  to  drive  T&E  requirements. 


•  Items  Being  Improved: 

-  Linkages  between  Warfighter  tasks  and  system  attributes. 

-  Reference  missions  and  tasks  and  mission/task  capabilities  standards. 

-  Procedures  to  roll-up  system  and  operational  performance  into  mission  accomplishment. 


Army  Proven 

Bottle  Ready 
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Desired  End  State 


Synchronized  with  Combat 
Developer. 

Synchronized  with  systems 
research,  development  and 
engineering. 


( - ; - \ 

Collaborative  environment  defined  by  a 

common  framework. 

V _ _ _ / 

Army  Proven 

Baffle  Ready 
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MBT&E  Point  of  Contact 

Christopher  Wilcox 

US  Army  Test  and  Evaluation  Command 
US  Army  Evaluation  Center 
ATTN:  TEAE-FI  (Mr.  Chris  Wilcox) 
4120  Susquehanna  Ave. 
Aberdeen  Proving  Ground,  MD  21005 


Army  Proven 

Battle  Ready 


Office:  (410)  306-2193 
chris.wilcoxl  @us.army.mil 
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“SV 

Defense  Information  Systems  Agency 

Department  of  Defense 


Ready  for  Scrum? 


Steve  Hutchison 

DISA  T&E 


A  Combat  Support  Agency 


Presentation  Tasks 


Backlog 


Scrum  Overview 


In  Progress 


A  Combat  Support  Agency 


Scrum  Overview 


Continuous  Integration  and  Test 

♦  Developmental  _ 

•  Operational 

•  Interoperability 

♦  Security 


Voice  of  the  User 


User  Stories 


Product 

Backlog 

(Requirements 

Generation) 


Sprint 

Backlog 

{Highest  Priority 
User  Requirements) 


Sprint 

(Capability 

Development) 


Sprint 

Review 

{Deployment 

Decision) 


Working  Product 

[Militarily  Useful  Capability) 


User  Requirements 


•  Software  development  framework  focused  on  delivering  value  according  to 
customer  priorities 

•  Delivers  working  software  with  fewer  defects  at  a  sustainable  pace 

•  Removes  impediments;  teams  self-orqanize  and  become  — h^er-productive” 


A  combat  support  Agency 


Why  “Scrum”? 


Tlsinew  emphasis  on  speed  and 


flexibility  calls  for  a  different  approach  '  n  ^ 

for  managing  new  product  development. 

The  traditional  sequential  or  =elay  race1 

approach  to  product  development... may  conflict 
with  the  goals  of  maximum  speed  and  flexibility. 

Instead,  a  holistic  or=ugby‘  approach — where  a 
team  tries  to  go  the  distance  as  a  unit,  passing 
the  ball  back  and  forth — may  better  serve  today‘s 
competitive  requirements.” 


Takeuchi  and  Nonaka,  — FTe  New  New  Product  Development  Game,” 


Harvard  Business  Review,  January  1986. 


A  Combat  Support  Agency 


“Agile  Manifesto” 


individuals  and 

interactions 

over 

processes 
and  tools 

working 

software 

over 

comprehensive 

documentation 

customer 

collaboration  over  contract 

negotiation 


responding 

to  change  over  following 

a  plan 

www.agilemanifesto.org 

— Wile  there  is  value  in  the  items  on  the  right, 
we  value  the  items  on  the  left  more.” 


A  Combat  Support  Agency 


Key  Features  of  Scrum 


Highly  collaborative 
Documentation  light 


Change  resilient 

Fundamentally  different 
requirements  process 

Short  duration  development  cycle:  — Spfit” 

Continuous  integration  and  testing 

Focused  on  priority  needs  of  the  customer:  the 
Warfighter 


Agile  is  about  delivery  of  capability  at  —peed  of  need.” 


A  combat  support  Agency 


3  Key  Roles  in  Scrum 


•  Product  Owner 

-  define  product  features 


-  prioritize  features;  adjust  at  each  sprint  — - 

-  accept/reject  sprint  product 

•  Team 

-  cross-functional,  self-organizing 

•  programmers,  users,  testers 

-  membership  does  not  change  during  sprints 

•  Scrum  Master 

-  enable  collaboration  across  all  roles  and  functions 

-  ensure  team  productivity  -  remove  impediments! 


Inspect  and  Adapt 


The  Product  Backlog 


A  Combat  Support  Agency 


•  Requirements  document 

-not a  CDD 


•  Prioritized  list  of  desired  features 

-  product  Owner  prioritizes 

-  stated  as  -user  story” 

•  as  a _ ,  I  want  to _ ,  so  that  I  can _ 

•  A  Mission  Thread  likely  consists  of  multiple  user  stories 

•  Continuously  updated  and  re-prioritized 

-features  added  and  removed  to  reflect  customer 


needs 


A  high-priority  user  requirement  may  be  just 
one  sprint  away  from  delivery 


The  Sprint 


A  Combat  Support  Agency 

•  Time-boxed  development  period 

-  design,  code,  test 

-  sustainable  pace 
- Spnit  backlog” 

-  highest  priority  features  from  product  backlog 

-  no  changes  (new  features)  during  Sprint 

•  Test  Driven  Development 

-  user  stories  translated  into  Test  Cases 

•  testing  the  capability  as  it  is  intended  to  be  used 

-  early  involvement! 


Outcome:  potentially  deployable  capability 


Continuous  Integration  and  Test 


A  Combat  Support  Agency 


Testing  is  a  shared  resource 

-  DT,  OT,  interoperability,  security 

-  continuous  user  involvement 

-  one  team! 

Reciprocity 

Risk-based,  mission  focused 

Maximizes  use  of  test  automation 

-  virtualization 

Lightweight  documentation:  shift  emphasis 
from  TEMP  to  Test  Cases 


RITE 

Rapid  Integration  ^  Test  Environment 


Do  not  sacrifice  rigor  in  Agile  testing 


Sprint  Review 


A  Combat  Support  Agency 


Deployment  decision 

-  not  an  FDDR 


•  Demo  to  stakeholders 

•  Working  capability  is  eligible  to  be  deployed 

-  product  owner,  stakeholders  decide 

-  can  be  improved  in  subsequent  sprint 

-  defects  returned  to  the  product  backlog 

•  Testers  take  on  — amtinuous  monitoring”  role 
for  deployed  capability 

Capability  deployment:  start  small,  scale  rapidly 


Task  Board 


A  Combat  Support  Agency 


Backlog 


In  Progress 


Role  of  Testing  in  Scrum 

A  Combat  Support  Agency 


Continu-,^  the  test  cases!  i  Tact 


User  Stories 


Sprint 


Testing  time  1 

in  estimation 
Complete  during 
Sprint  timebox? 


Rinse 

Repeat 


'  resources 
'  users 
'  tools 


“  Requ 


forDoD 


Jabilitj 


^lai itatii 


Oils 


Working  Product 

lit  aril1 

C°ntinuoUs 
Monitoring 

performance  (g) 

scale ? 


If  you  think  you're  going  to  show  up  at  the  end  and  run  a  test,  think  again. 


Task  Board 


A  Combat  Support  Agency 


Backlog 

In  Progress 

Done 

Role  of  Testing  in 

Scrum 

Scrum  Overview 

1 _  | 

Agile  Testing 

Summary 

A  combat  support  Agency 


What  is  Agile  Testing? 


Is  not  testing  on  an  Agile  project 
— Ediy  Tester  Involvement” 


-  drive  development 

•  One  team  -  no  tester  silos 

-  customer  focused 

•  Collaborative 

-  with  developer,  customer 

-  not  — qu%  police” 

— blfortunately,  customers  aren't  generally  good  at  articulating  their 
requirements.  Driving  development  with  the  wrong  tests  won't 
deliver  the  desired  outcome. 


Crispin  and  Gregory,  Agile  Testing 


A  combat  support  Agency 


What  Makes  Agile  Testing 


Different? 


•  Not  a  “phase”  at  the  end 

-  Can't  test  quality  into  the  product 


•  Much  earlier  in  the  process  U. 

-  Coding  and  testing  are  integrated 

•  A  user  story  is  not  — ctoe”  until  it  has  been  tested 


-  Drives  a  culture  of  feedback  and  improvement 

•  Not  a  gatekeeper 

•  Lightweight  process 

-  Less  documentation  reliant 

•  Employs  more  automation 

We  define  an  agile  tester  this  way:  a  professional  tester  who 
embraces  change,  collaborates  well  with  both  technical  and 
business  people,  and  understands  the  concept  of  using  tests  to 
document  requirements  and  drive  development. 


Crispin  and  Gregory,  Agile  Testing 


Agile  Testing  Quadrants 


E 

TO 

|S 

(Li 


c n 
c 

'-E 

o 

Q. 

Q. 

D 

CO 


Functional  lests 
Examples 
Story  Tests 
Prototypes 
Simulations 


Exploratory  Testing 
Scenarios 
Usability  Testing 
U AT  (User  Acceptance  Testing) 
Alpha  /  Beta 


Q2  Q3 
Q1  Q4 


Unit  Tests 
ComponentTests 


Performance  S  Load  Testing 
Security  Testing 
"ility"  Testing 


Technology  Facing 


Copyright  2009  Lisa  Crispin, 

Janet  Gregory  -  DragonFire,  Inc. 

The  Agile  Testing  Quadrants 

Original  idea  by  Brian  Marick,  www.exampler.com 


Critique  Product 


Task  Board 


A  Combat  Support  Agency 


Backlog 


In  Progress 


Done 


Scrum  Overview 

i _ 


Role  of  Testing  in 
Scrum 


A  Combat  Support  Agency 


Challenges 


•  Test  as  a  Service 

-  On  demand 

•  Persistent  environment 

-  Federate  capabilities 

-  Virtualize 


•  Education  and  training 

-  PMs  and  Testers 


•  Agile  DIACAP,  Interoperability,  oversight 

•  Community  of  Interest  User  base 


Shift  the  paradigm:  testing  as  an  enabler 
of  improved  acquisition  outcomes 


A  combat  support  Agency 


Summary 


New  IT  Acquisition  model  is  coming 


•  TE&C  processes  must  adapt  — 

-  Responsive  to  iterative,  incremental  development 

-  Responsive  to  User  priorities 

-  High  optempo 

•  Dramatically  reduced  TE&C  timelines 


Objective:  rapid  fielding  of  enhanced  IT 


capabilities  to  the  Warfighter 


Task  Board 


A  Combat  Support  Agency 


Backlog 

In  Progress 

Done 

Scrum  Overview 

\ 
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Role  of  Testing  in 
Scrum 

Summary 
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Expanded  Use  of  the  Probability  of 
Raid  Annihilation  (P^)  Testbed 


Introduction 

The  PRA  testbed  support  to  LPD-17  is  complete,  but  the  testbed  itself  is  just 
maturing  into  a  multi-use  toolset  .  This  toolset  has  possibilities  of  supporting 
other  test  events/systems  outside  of  the  confines  of  the  current  planned  events. 
In  order  to  realize  and  capitalize  on  the  engineering  research,  development  and 
subsequent  refinements  in  the  testbed  it  is  well  worth  exploring  all  potential 
avenues  for  employing  this  federation  of  models. 


Presentation  Outline 

-  PRA  and  the  Enterprise 

-  Capitalization 

-  Areas  for  Expansion 

-  Fleet  Tactics  Development 

-  S/W  Validation 


-  SSTAMDA  vs  AAW  CRD 

-  Trade  Studies 

-  Evolving  Capabilities  and  Threats 

-  Predictive  Analysis 

-  Conclusion 
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PRA  and  the  ENTERPRISE 


•  Probability  of  Raid  Annihilation  (PRA)  Testbed  forms  one  of 
the  core  elements  of  the  Enterprise  Approach  to  Anti-Air 
Warfare  Ship  Self  Defense  (AAW  SSD). 

•  Proven  capability  to  provide  answers  for  LPD  17  AAW  SSD 
MOE  and  other  Enterprise  platforms/systems. 


•  Capability  exists  to  answer  other  questions,  to  further 
expand  the  use  of  the  testbed. 


Physics-based 

Model 


Tactical 

SWIL/HWIL 


JHU  Applied  Physics  Lab 
,  Laurel,  MDr 


Naval  Research  Lab 
Washington,  I 


NA  WC  Weapons  Division 
China  Lake  .  \ 


Scenario  & 
Environment 
Federate  (SEF) 


Virtual  Range 
Instrumentation: 
SIMDIS, 
HLAResults 


Background 

Targets/ 

Emitters 


1  SLQ-32 

1 

SPS-48E 

SPQ-9B 

■  1 

Network  Interface  Layer  -  RTI NC  Pro 


L 


RAM  r  ram 

Missile  ■  Launchers 
Salvo  " 


t  Threat 
Types 


reactive  multi-threat  raid 


Ship  Motion  ■  D 
&  Signatures  ■  Uecoys 


SSDS 

CEP 

S/W  Release 
Validation 


Predictive 

Analysis 


Evolving 
Capabilities 
and  Threats 


Future  Upgrade/Expansion  -  Capitalize  on  today's  foundation 
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”“,ES  Capitalization 

•  Capitalize  on  engineering  research  &  development 
completed  to  date: 

•  Operationally  relevant  environments 

•  Common/shared  geographies,  threats,  and  weapons 

•  Sensor  performance 


“Clean” 

and 

“Dirty” 

Signatures 


•  Common  Lethality  Federate  (CLF) 


Scenario 


T1R1  -  sea-skimming,  subsonic  RF  threat 

T2  -  sea-skimming,  subsonic  Imaging  IR  threat 

T5  -  high  diver,  supersonic  RF  ARM  threat 

T7  -  sea-skimming,  maneuvering  supersonic 
Advanced  RF  Threat 
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•  Six  Preliminary  Areas  for  consideration  in  Testbed 
Expansion 


Virtual  Range 
Instrumentation: 
SIMDIS, 
HLAResults 


Background 

Targets/ 

Emitters 


Network  Interface  Layer  -  RT!  NG  Pro 


Key: 


Physics-based 

Model 


Ship  Motion 
&  Signatures 


Decoys 


Environment 
Federate  (SEF) 


/ 


SPS-48E  I  SPQ-9B 


SSDS  ■  CEP 


•  These  areas  represent  beginning  possibilities. 

•  As  other  testbeds  are  developed,  co-use  opportunities  arise 
-  example  is  DDG  1000  MOE  Testbed. 
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•  Fleet  Tactics 

•  Currently  developed  using  intelligence,  threat  assessments,  M&S,  and 
tactician  analysis  validated  by  limited,  if  any,  formal  live  end-to-end  test 
events. 

•  Testbed  can  used  to  verify  interim  and  experimental  tactics  techniques  and 
procedures  against  hi-fi  representations  of  appropriate  threats  in  end-to-end 
fashion. 


•  Ability  to  develop  and  verify  tactics  via  modeling  and  simulations  (M&S) 
affords  the  opportunity  to  vary  the  tactical  environment. 
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S/W  Validation 

Validation  of  Software  Releases 

•  Testbed  could  be  used  to  test  and  validate  tactical 
software  updates,  thereby  eliminating  the  need  to  test  and 
retest  on  fleet  units  or  test  facilities. 

•  Ties  up  fleet  assets 

•  Scarce  test  resources 

•  An  invaluable  tool  to  mitigate  time  and  costs 

•  Saves  test  resource  usage 
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SSTAMDA  VS  AAW  CRD 


•  SSTAMDA  imposes  PRA  requirements  on  future  ship 
classes  &  and  existing  ships  with  significant  combat 
system  upgrades. 

•  Recommended  PRA  for  existing  ship  classes  are  different 
from  AAW  CRD. 


•  Future  ship  classes  are  required  to  demonstrate  they 
meet  the  new  PRA 


•  The  Testbed  easily  adaptable  to  test  virtual  ship 
against  the  new  requirements. 


Scenario  & 
Environment 
Federate  (SEF) 


Virtual  Range 
Instrumentation: 
SIMDIS, 
HLAResults 


Network  Interface  Layer  -  RT!  NG  Pro 


Physics-based 

Model 


Ship  Motion 
&  Signatures 


Decoys 


New  Requirements  tested  in  a  Proven  Fashion 


Key. 


SPS-48E  I  SPQ-9B 


SSDS  ■  CEP 
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Trade  Studies 


•  Use  of  the  Enterprise  Testbed  to  determine: 


•  If  a  particular  combat  system  configuration  will  perform 
successfully  to  meet  PRA  requirements. 

•  Efficacy  of  future  ship  class  combat  systems  variations 
investigated  to  an  increased  degree  of  accuracy. 

•  Prior  to  full  funding  commitment  or  design. 


•  Cost  saver. 
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technologies  Evolving  Capabilities  and  Threats 

•  Testing  combat  systems  against  evolving  threats 

•  With  low  probability  of  intercept  radars. 

•  With  coherent  radars. 

•  With  low  observable  technologies. 

•  Navy’s  advancement  with  (SEWIP) 

Block  II AN/SLQ-32  B,  will  require: 

•  Higher  fidelity  emissions  from  threat  surrogates 
(targets)  and  threat  models. 

•  Use  of  the  Testbed  would  realize  cost  savings  and  fill 
shortcomings  imposed  by  a  lack  of  capable  targets. 
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•  Changes/updates  to  the  appropriate  model(s)  in  the 
Testbed  would  facilitate  a  sufficient  number  of  Monte- 
Carlo’ed  scenarios  for  pre-flight  prediction. 

•  Provides  ability  to  craft  and  run  a  number  of  scenarios  to 
find  the  heart  of  &  limits  of  the  test  envelope. 

•  Combine  with  Design  of  Experiments  to  define  the  test 
space  and  number  of  live  events  to  achieve  statistically 
significant  results. 
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Expanded  Use  of  the  Probability  of  Raid 
Annihilation  (P^)  Testbed 
Conclusion 


•  Capability  exists  to  answer  other  questions,  to  expand 
the  use  of  the  testbed. 

•  The  initial  development  is  complete  and  proven  through 
the  LPD  1 7  PRA  effort. 

•  A  variety  of  efforts  would  benefit  from  a  complementary 
M&S  input: 


•  Fleet  TTP 

•  Software  validation 

•  SSTAMDA 

•  Trade  Studies 


•  Evolving  Capabilities  and 
Threats 

•  Predictive  Analysis 
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Expanded  Use  of  the  Probability  of  Raid 
Annihilation  (PRA)  Testbed 
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Design  of  Experiments 

"Managing  Expectations" 
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Agenda 


"View  from  the  trenches" 

Why  test,  Why  learn? 

Why  DOE  makes  sense 

Manage  Expectations  -  What  works  (for  us) 

Questions? 
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Why  Test? 


-  Why  Test? 

-To  learn  and  bound  capabilities 
-  To  answer  some  basic  questions 
-Does  system  meet  capability 
requirements? 

-  What  is  actual  system  performance? 
-How  is  system  best  employed?  (Tactics, 
Techniques  and  Procedures) 
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Why  Learn? 


-  Why  learn? 

-  To  discover  the  "truth"  as  best  we  can  know  it 

-  To  enable  knowledgeable  proRram  decisions 


Guidance 
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-  Mandated  use  in  Gov't  T&E 

-  DOT&E  requires  DOE  in  Operational  Testing 

-  Recent  DDT&E  guidance  on  Developmental 


Testing 

-  Service  OTAs  have  Joint  MOA  naming  DOE  as  a 


r lv  nm/Luui  r.S 


Why  DOE? 


Scientific  Answers  to  Four  Fundamental 

Test  Challenges 


Four  Challenges  faced  by  any  test 


1.  How  Many?  A:  Sufficient  samples  to  control  our  twin  errors  -  false  positives  &  negatives 

2.  Which  Points  and  What's  Good?  A:  Span  the  battle-space  with  orthogonal  run  matrices 
using  continuous  measures  tied  to  the  test  objectives 

3.  How  to  Execute?  A:  Randomize  and  block  runs  to  exclude  effects  of  the  lurking, 
uncontrollable  nuisance  variation 

4.  What  Conclusions?  A:  Build  math-models*  of  input/output  relations  (transfer  function), 
quantifying  noise,  controlling  error 


Noise 


Design  of  Experiments  effectively 
addresses  all  these  challenges! 


Inp 

(X 


Noise 


Outputs 

(Y’s) 


Many  model  choices:  regression,  ANOVA,  etc. 


AVW 


Tester's  Challenge 


-Time  to  execute  the  test 

-  Resources  to  support  the  full  scope  of  planned  test 

-  Funding 


DOE  Test  Process: 

Well-Defined  From  Blank  Paper  to  Conclusions 


Planning:  Factors 
Desirable  and  Nuisance 
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CL-  +0  38 

+0.26  x  A-o-A 

+0.0T7  x  Sideslip 

+0.081  X  Stabilizer  Deflection 

-.00376  x  LEX  Type 
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0.315 
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Difference 

Here 


How  to  Execute 


Four  Stages 

■  Plan  deliberately: 
problem, 

objective(s),  outputs, 
inputs,  background 
variables,  phases 

Design  for  power  in 
spanning 

battlespace:  many 
choices  of  designs, 
depends  on  your 
system 

■  Execute  with 
insurance  against 
lurking  variables  and 
unknown-unknowns 

Objectively  analyze 
with  statistical 
methods 

(ANOVA/Regression) 
to  determine  what 
matters,  direction, 
magnitude 
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Why  DOE  Makes  Sense 


DT&E:  Science  &  Engineering  are 
Vital  to  Success  of  our  Tests 


We  already  have  good  science  in  our  DT&E! 

We  understand  sys-engineering,  guidance,  aero, 
mechanics,  materials,  physics,  electromagnetics  ... 

DOE  introduces  the  Science  of  Test 
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Why  DOE  Makes  Sense 


OT&E:  Operations  Skills  are  Vital 
to  the  Success  of  Test 


Similarly:  we  already  have  good  ops  in  our  OT&E! 

We  understand  attack,  defense,  tactics,  ISR,  mass, 
unity  of  command,  artillery,  CAS,  ASW,  AAW,  armored 

cav... 

DOE  adds  the  Science  of  Test 


We  make  decisions  too  important  to  be  left  to  professional  opinion 
alone... our  decisions  should  be  based  on  mathematical  fact 


Greg  Hutto 
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Managing  Expectations 


Observation  by  a  Practitioner 

-  At  this  point  in  history,  (for  OT)  using  DOE  simply 
means  laying  out  the  primary  factors  that  affect  the 
response  variable  in  at  worst  a  notional  design  (and  at 
best  a  design  that  one  could  readily  use  with  proper 

resources  and  leadership  support) 


Dr.  R.  McIntyre  Feb  2011 
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What  Works  (for  us) 


• DOE  provides  for  efficient  testing  and  more  useful  results  -  but  not 
necessarily  at  a  reduced  upfront  cost 

• DOE  is  most  effectively  applied  early  in  the  development  process 
where  build  a  little ,  test  little  is  cost  effective 
• Know  your  process;  know  the  tool 

• Investing  the  time  up  front  for  process  decomposition  (MBTD/E) 
will  pay  great  dividends  in  developing  the  experimental  design 
• Use  a  DOE  practitioner  to  assist  in  the  actual  design  development 
(then  execute  the  design) 

• Clearly  articulate  the  pros  and  cons  of  each  design  (metrics 
scorecard) 

• Ask  better  questions  ;get  better  answers 
• Even  when  DOE  is  not  the  correct  tool  to  use  for  a  particular 
application ,  it  will  at  least  aid  you  in  discovering  the  most  useful 
demonstrations  to  observe  (May  need  to  use  other  DOE- 1  ike  tools  - 
HTT) 
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QUESTIONS? 


BACK-UPS 


DOE  Metrics  Scorecard 


Basic  Report  Card  -  Designed  Experiments 

Wheel 

Design  Alternative 

0 

1 

2 

3 

Design  Name 

Baseline 

CCD  x3Cat 

2A5+4cp 

2A5-l+4cp 

Number  of  Factors 

Levels  ea  Factor 

Plan 

Num  Responses  (MOPS) 

Real-values? 

Objective? 

Test  Events  (N) 

Savings  (-Incr) 

n  •  * 
Design 

jjvliasing/Res/Ortho/Conf 

ound 

comparisons) 

2  a  Power 

Name  Design  Strategy 

Randomized? 

Execute 

Blocked  or  calibrated? 

Replicates?  True? 

Pred  Model  Supported 

Analyfl 

FDS  Pred  Err  @50/95% 

.Leverage  Avg/Max 

11 

SI/IF  Avg/Max 

DOE  expert  assistance  recommended 


Aerial  Tgts  Example 


Aerial  Target  Report  Card  -  Designed  Experiments 

Design  Alternative 

0 

1 

2 

3 

wneei 

Design  Name 

Baseline 

Factorial 

2A(6-l)x3 

7v  2/3  D  Opt 

Number  of  Factors 

3 

3 

7 

7 

Levels  ea  Factor 

2x2x3 

2x2x3 

2,3 

2,3 

Plan 

Num  Responses  (MOPS) 

1 

1 

1 

1 

Real-values? 

no 

no 

no 

no 

Objective? 

no 

no 

no 

no 

Test  Events  (N) 

13 

12 

96(12) 

46(6) 

Savings  (-Incr) 

— 

8% 

8% 

54% 

pVliasing/Orthogonality 

Res  II  (A=B) 

Full  Res 

RV+ 

Lseaign 

comparisons) 

5% 

5% 

5% 

5% 

2  a  Power 

5-65% 

50-82% 

99.90% 

99% 

n 

Name  Design  Strategy 

?? 

Factorial 

:ractionxCal 

Dopt  Fract 

Randomized? 

— 

— 

— 

— 

Execute 

Blocked  or  calibrated? 

— 

— 

— 

— 

Replicates?  True? 

- 

Pred  Model  Supported 

Main  Eff 

3  FI 

3FI 

2FI 

FDS  Pred  Err  @50/95% 

.72/1.1 

.71/.71 

.33/.42 

.66/.  77 

.Leverage  Avg/Max 

.38/1 

.5/.5 

.375/375 

.37/.47 

It^/IFAvg/Max 

2/2.5 

1/1 

1/1 

1.2/1. 3 

Summary  thoughts  ...  avoid  binary,  define  test  event,  max  events  per 
sortie/mission,  create  design  alternatives,  exploit  sequential 
experimentation 
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Analysis  Purpose  and  Approach 
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Issue: 

1 .  Current  body  armor  test  procedures  do  not  quantify  the  potential  threat 
to  the  Soldier  from  reflective  spall. 

-  Reflective  Spall  is  fragmenting  debris  that  results  when  a  threat  is 
defeated  by  the  protective  plates  in  body  armor. 

2.  Current  Army  operating  procedures  mandate  that  a  plate  must  be  turned 
in  if  the  ceramic  tile  is  exposed.  This  means  that  any  small  rip  in  the 
protective  nylon  cover  renders  a  plate  unserviceable. 

Problem: 

1 .  Is  there  a  risk  of  injury  to  the  Soldier  from  reflective  spall? 

2.  Is  the  risk  of  injury  higher  when  the  ceramic  tile  is  exposed? 

Approach: 

•  Perform  custom  experimental  testing  to  collect  fragmentation  using 
ballistic  gelatin. 

•  Conduct  personnel  vulnerability  modeling  and  analysis  using  MUVES-S2. 
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Experimental  Configuration 


Shot  Locations 


Target: 

•  Medium  protective  plate  contained  in  a  small/medium  shoot-pack 
Threat: 

•  Small  caliber  projectile 
Test  Matrix: 

•  20  shots  were  fired  at  each  shot  location 

•  1 0  with  the  nylon  cover  on 

•  1 0  with  the  nylon  cover  off 
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For  the  experimental  testing  a  custom  target  was  designed  to  collect  all  of  the  fragmentation.  The  target  consisted  of  a 
medium  size  protective  plate  with  ballistic  gelatin  surrounding  it.  The  gelatin  blocks  provided  backing  for  the  target,  a  medium 
size  protective  plate,  as  well  as  a  witness  collection  medium  for  reflective  spall  that  could  potentially  impact  a  Soldier’s  arms, 
legs,  and  head. 


The  distances  between  the  plate  and  the  gelatin  were  determined  using  the  digital  human  reference  anatomy  used  in  ARL’s 
Operational  Requirement-based  Casualty  Assessment  (ORCA)  model. 
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Personnel  Injury  Analysis 


All  fragments  recovered  in  the  ballistic  gelatin  were  evaluated. 


•  For  each  observed  occurrence,  the  fragmentation  was  characterized  in  terms  of  velocity, 
mass,  shape  factor,  trajectory,  entrance  point,  stopping  point,  density,  and  material  type. 


•  The  ORCA  skin  penetration  equation  was  used  to  filter  out  fragments  with  less  than  a  50% 
probability  of  skin  penetration.  These  fragments  will  cause  a  superficial  injury  or  no  injury 
at  all. 


•  The  angle  the  fragment  traveled  was  used  to  evaluate  if  the  fragment  could  potentially  hit 
a  body  region. 

•  MUVES-S2  flew  the  fragment  in  3-dimensions  into  the  3-dimensional  ORCA  human 
anatomy. 

•  ORCA  modeled  the  permanent  wound  cavity  of  each  fragment  and  scores  the  severity  of 
each  injury. 


The  Maximum  Abbreviated  Injury  Score  (MAIS)  was  used  to  determine  the  likelihood  of  a 
significant  injury. 
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Analysis  Injury  Metric:  MAIS 


*4 


MAIS  -  Maximum  Abbreviated  Injury  Score  (MAIS)  is  an  anatomical  measure  of  injury  severity.  This  score  classifies  injury  severity  on  the 
basis  of  the  single  injury  having  the  greatest  AIS  severity  value.  The  MAIS  is  between  0  and  6. 

Serious  Injury  -  An  injury  that  requires  immediate  medical  attention  and  the  threshold  criteria  for  significant  injury  for  this  analysis. 
Untreated  serious  injuries  could  deteriorate  and  cause  loss  of  life. 


MAIS 

Injury 

Level 

Head  Injury  Example 

Type  of  Injury 

1 

Minor 

Minor  laceration  of  scalp 

Superficial 

2 

Moderate 

Major  laceration  of  scalp,  blood  loss 
<  20% 

Reversible  injuries;  medical  attention 
required 

3 

Serious 

Fracture  of  skull,  penetration  <  2  cm 

Reversible  injuries;  hospitalization 
required 

4 

Severe 

Depressed  skull  fracture,  penetration 
>  2  cm 

Life  threatening;  not  fully  recoverable 
without  care 

5 

Critical 

Depressed  skull  fracture,  laceration 
of  spinal  artery 

Non-reversible  injuries;  not  fully 
recoverable  even  with  care 

i 

“Abbreviated  Injury  Scale”  (AIS)  is  an  anatomically-based,  consensus-derived,  international  severity  scoring  system  that  classifies  each  injury  by  body  region  according  to  its  relative 
importance  on  a  6-point  ordinal  scale.  AIS  values  provide  information  on  the  type,  location,  and  severity  of  anatomical  injuries.  AIS  scores  each  single  injury. 

*AIS  is  copyrighted  by  the  Association  for  the  Advancement  of  Automotive  Medicine  (AAAM),  Barrington,  IL 
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zxample  of  Personnel  Injury  Analysis  for  the  Head  Region 


Ballistic  Experimentation 


Head 


E 

1— 

< 


4- 

Recovered  Fragments 


Fragment  Characteristics: 

•  Mass 

•  Shape  Factor 

•  Impact  Point 

•  Stopping  Point 

•  Material  Type 

•  Material  Density 

•  Striking  Velocity 


Filtering 

Velocity,  Mass,  and  Shape  Factor 

a 

ORCASkin 
Penetration 
Equation 


Probability  of  Skin  Penetration 

4 


Feasibility  of 
Fragment 
Trajectory 


Fragments  Used  in  Evaluating  Injury  Risk 


Analysis 


■ 


MAIS 


2  3  4 


Probability 

Calculation 


Probability  of  MAIS  3  or  Greater 
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™  kdecom^  ^  Analysis  of  Skin  Penetrating  Fragments 


Recovered  Fragments 


Material  Composition  of  Skin  Penetrating  Fragments 


□  Fragments  from  Copper  Jacket  □  Fragments  from  Steel  Penetrator  □  Fragments  from  Plate 


100% 

90% 

80% 
x  70% 

=  60% 

0) 

=  50% 

|  40% 

30% 

20% 

10% 

no/ 

Fragment  Mass  Distribution 

77.8% 

17.6% 

3.1% 

0.6%  0.5%  0.2%  0.1%  0.1%  0.1% 

u  /  0 

<bN  <o)'  "O'  <£>  C?  <§>  <&  V1  K<&  nC>N  n'bN  'b' 

Or  O*  Qy  v  V  v  v  v  <v  *V  JjV' 

Mass  (Grains) 

Striking  Velocity  Distribution  of  Skin  Penetrating  Fragments 


Velocity  (m/s) 
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Probability  of  Injury  for  Shot  Location  One 


Average  P(MAIS>=2)  for  Location  1 


100%  j 

90%  - 

80%  - 
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II 

70%  - 

CO 

< 
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oT 
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0) 

U) 

CO 

40%  - 

i- 

0) 
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Impact  of  Nylon  Cover  on  the  Number  of  Fragments  (Location  1) 


Q) 

Si 

E 

3 


300 

250 

200 

150 

100 

50 

0 


i  Non  Skin  Penetrating 
Fragments 
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No  Nylon 
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No  Nylon 

Cover 
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Nylon  Cover 
i  No  Nylon  Cover 
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Average  P(MAIS>=3)  for  Location  1 
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™RDECO!\nm  probability  of  Injury  for  Shot  Location  Two 


Impact  of  Nylon  Cover  on  the  Number  of  Fragments  (Location  2) 
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i  Non  Skin  Penetrating 
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Skin  Penetrating 
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tyRDECOMm  probability  of  Injury  for  Shot  Location  Four 


Impact  of  Nylon  Cover  on  the  Number  of  Fragments  (Location  4) 
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▼  rdfcoivi  >  Conclusions 


Is  there  a  risk  of  injury  to  the  Soldier  from  reflective  spall? 

•  42%  of  all  recovered  fragments  had  a  probability  of  skin  penetration  less  than  50%. 

•  Of  skin  penetrating  fragments 

-  77%  have  masses  between  0.15  grains  and  0.31  grains 

-  95%  have  a  striking  velocity  less  than  400  m/s 

•  Fragments  impacting  under  the  chin  had  the  highest  probability  of  MAIS  3  or  greater  and 
that  probability  was  <  2%  from  all  shot  locations. 

•  Protective  plate  reflective  spall  is  highly  unlikely  to  cause  a  serious  or  greater  injury  to  the 
Soldier. 


Is  the  risk  of  injury  higher  when  the  ceramic  tile  is  exposed? 

•  Of  skin  penetrating  fragments  <  1  %  from  the  plate  material  (99.2%  from  the  threat) 

•  Typically  a  larger  number  of  fragments,  and  skin  penetrating  fragments,  are  produced  from 
the  plates  without  nylon  covers  than  the  plates  with  nylon  covers. 

•  No  considerable  difference  in  probability  of  serious  or  greater  injury  with  or  without  the 
nylon  cover. 

Conclusions: 

1 .  Soldiers  are  not  at  risk  from  reflective  spall  fragments. 

2.  Exposing  the  ceramic  tile  does  not  increase  the  risk  of  injury  from  reflective  spall  fragments. 
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Analytical  Approach  using  MUVES-S2/ORCA  Modeling 
in  Support  of  the  Joint  Cargo  Aircraft  (JCA) 

National  Test  and  Evaluation  Conference 
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Richard  Moyers  and  Penny  Willard 

U.S.  Army  Research  Laboratory 
Survivability/Lethality  Analysis  Directorate 


•  Background 

•  ComputerMan/Pilot  Survey  (CMPS)  Methodology 

•  Operational  Requirement-based  Casualty  Assessment  (ORCA)  Methodology 

•  Job  Description  Development 

•  Joint  Cargo  Aircraft  (JCA)  Jobs 

•  Mission  Scenarios 

•  ORCA  Personnel  Inputs 

•  JCA  Vulnerability  Analysis 

•  Conclusion 
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•  Ballistic  vulnerability  analysis  was  a  joint  effort  between  the  U.S 
Army  Research  Laboratory,  Survivability/Lethality  Analysis 
Directorate  (ARL/SLAD)  and  the  U.S.  Air  Force  Aeronautical 
Systems  Center  Engineering  Directorate  (ASC/ENDA) 

•  Two  analyses  were  conducted: 

-  Crew  vulnerability  analysis 

-  System-level  vulnerability  analysis 

•  Personnel  configurations: 

-  4  Crew  Members:  pilot,  copilot,  two  loadmasters 

-  44  Crew  Members:  pilot,  copilot,  two  loadmasters,  40  troops  in  cargo  area 


3 


TECHNOLOGY  DRIVEN.  WARFIGHTER  FOCUSED. 


-  US  ARMY 

0  RDEConnm 


CMPS  Methodology 


•  ComputerMan/Pilot  Survey  (CMPS) 

-  Evaluation  methodology  combined  Computer  Man"s  limb  state  performance 
with  Pilot  Survey  results  to  predict  residual  functionality  for  an  injured  crew 
member 

>  ComputerMan 

•  Based  on  limb  dysfunction 

•Assessed  wound  tract  size  and  tissue  retardation 

•  Evaluated  for  4  representative  combinations:  Defense  -  30  seconds, 
Assault  -  30  seconds,  Assault  -  5  minutes,  and  Supply  -12  hours 

•  Divided  model  into  81  Functional  Groups  (FGs) 

•Assessed  against  3  levels  of  dysfunction:  none,  partial  and  total 

>  Pilot  Survey 

•  Used  to  predict  the  crew"s  ability  to  continue  to  operate  the  aircraft 
after  sustaining  various  levels  of  ballistic  injury. 


-  For  aviation  analyses,  “Assault  -  30  seconds”  data  was  used  to  assess  pilot 
and  co-pilot  residual  functionality 
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ORCA  Methodology 


Battlefield 

Insult 


0 


User 

Input 


Trauma  /  Injury 
Characterization 


Medical 

Assessment 


Survivability  /  Protection 
Effectiveness 


Medical 
Casualty? 


Survivability  Metric  (Injury) 

Typically  Red  on  Blue  analyses 

Focus  is  the  survivability  of  Blue  forces 

ORCA  output  metric:  MAIS 

Example  analysis:  Vehicle  up-armor  programs 


“1 


Individual 

Characterization 


Military  Task 
Requirement 


Elemental 
Capability 
Impairment 


Elemental 
Capability 
Requirement 


Capability 

vs. 

Requirement 


Operational  Casualty? 


Lethality  Metric  (Incapacitation) 

Typically  Blue  on  Red  analyses 
Focus  is  the  lethality  of  a  Blue  system 
ORCA  output  metric:  Operational  Casualty 
Example  analysis:  Munition  lethality  evaluation 
1 .  WARFIGHTER  FOCUSED. 


Job  Description  Process 

1 .  Identify  the  functional  tasks  that 
are  likely  to  be  called  upon  during 
the  context-sensitive  scenario. 

1 .  Define  the  scenario 

2.  Confine  the  task  list  to  the 
scenario! 

2.  Cull  doctrinal  /  training  manuals  to 
decompose  the  tasks  into  their 
unique  and  discrete  task  elements 
(example:  to  get  into  the  car,  you 
must  first  open  the  door,  followed 
by  turning  to  seated  position,  and 
then  close  the  door). 

3.  Consult  SMEs  and  practical 
experience  to  quantify  each  task 
element  into  minimum  (threshold) 
and  full  (objective)  levels  of 
performance  within  the  Elemental 
Capability  Vector  (ECV). 

4.  Pursue  verification  and 
endorsement  from  SMEs. 


Elemental  Capability  Vector  * 


Visual  Acuity  &  Color 
Discrimination 

Speech  Articulation 

Night  Vision 

Vocal  Power 

Visual  Field  of  View 

Right  Leg  Strength 

Visual  Binocularism  & 

Motility 

Left  Leg  Strength 

Hearing  Threshold  -  Low 

Freq. 

Left  Arm/Hand  Strength 

Hearing  Threshold  -  High 

Freq. 

Right  Arm/Hand  Strength 

Binauralism 

Left  Arm/Hand  Dexterity 

Endurance 

Right  Arm/Hand  Dexterity 

Psychomotor  Mental 

Processing 

Torso  Support 

Cognitive  Mental  Processing 

Head  /  Neck  Movement 

Visual  Mental  Processing 

Somatic  Senses 

Auditory  Mental  Processing 

Balance 

*not  in  order 
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Job  Decomposition 


A  job  is  defined  as  a  list  of  an  individual’s  tasks. 


A  task  is  made  of  task  elements.  A  job  can 
have  one  or  more  tasks. 


A  task  element  is  a  single  activity  with 
specific  parameters.  An  instance  of  a  task 
element  is  defined  in  the  terms  of  elemental 
capabilities. 
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Example  Job 

Assaulting  Infantry  Rifleman 


m 


Engage  Targets  with  an  M4  or  M4A1  Carbine 
Load  an  M4  or  M4A1  Carbine 

Mount  a  NVS  AN/PVS-4  on  a  M4/M4A1  Carbine 
Zero  a  NVS  AN/PVS-4  to  a  M4/M4A1  Carbine 
Engage  Tgts  w/  M4/A1  Carbine  Using  NVS  AN/PVS-4 

Mount  an  AN/PAS-13  TWS  on  M4/M4A1  Carbine 
Zero  an  AN/PAS-13  TWS  to  an  M4  or  M4A1  Carbine 


Engage  Tats  w /  M4/A1 /Carbine  Using  NVS  AN/PVS-4 

-  With  sight  in  operation  assume  appropriate  firing 
position  based  on  situation 

-  Identify  targets  in  designated  sector  of  fire 

-  Determine  range  to  targets  using  the  AN/PVS-4 
reticule 


Engage  Targets  with  an  M4/M4A1  Carbine  Using  a  TW  -  Fire  on  target  until  destroyed  or  told  to  cease  fire 


Engage  Targets  w/  M4/M4A1  Carbine  Using  an  AN/PAS-13 


Operate  a  Night  Vision  Sight  AN/PVS-8 

Operate  a  Thermal  Viewer,  AN/PAS-7 
Perform  Safety  Checks  on  Hand  Grenades 
Employ  Hand  Grenades 
Engage  an  Enemy  with  a  Bayonet 
Move  as  a  Member  of  a  Fire  Team 


Employ  Hand  Grenades 

-  Position  body  remaining  covered 

-  Grip  grenade  with  lever  down  and  pull  ring  free 

-  Arm  grenade  by  removing  safety  clip  and  ring 

-  Confirm  body  alignment  and  keep  eyes  on  tgt 

-  Throw  grenade  overhand  with  eyes  on  target 

-  Return  to  cover  and  concealment 

WARFIGHTER  FOCUSED. 
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Elemental  Capability  Vector  * 

Visual  Acuity  &  Color  Speech  Articulation 

Discrimination 


Night  Vision 

Visual  Field  of  View 

Visual  Binocularism  & 
Motility 

Hearing  Threshold  -  Low 
Freq. 

Hearing  Threshold  -  High 
Freq. 

Binauralism 

Endurance 

Psychomotor  Mental 
Processing 

Cognitive  Mental  Processing 
Visual  Mental  Processing 
Auditory  Mental  Processing 


Vocal  Power 

Right  Leg  Strength 

Left  Leg  Strength 

Left  Arm/Hand  Strength 

Right  Arm/Hand  Strength 

Left  Arm/Hand  Dexterity 

Right  Arm/Hand  Dexterity 

Torso  Support 

Head  /  Neck  Movement 

Somatic  Senses 

Balance 


Elemental  Capability  Value  Acquisition  Lexicon 


Maximum 


Full 

Each  EC  is  described 
in  both  the  minimum 
and  full  level  of 
performance  within 
the  ECV. 


Minimum 


Threshold 
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Pilot  /  Co-Pilot:  The  two  people  charged  to  jointly  fly  the  aircraft. 

*  Assume  redundancy  of  functions. 

-  Airborne  through  Landing 

-  Descent  through  Landing 


GIBs:  Personnel  in  the  aircraft  cabin:  loadmasters  and  troops 

(aka  “Guys  in  Back”) 

-  GIB  Egress 
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What  Assumptions  Did  We  Make 
About  the  JCA  Jobs? 


Due  to  the  assumption  of  redundancy,  these  jobs  were  built  by 
reversing  the  direction  of  the  traditional  ORCAjob  architecture. 

Traditionally,  a  Job  is  mapped  to  a  person. 

Vehicle  Driver  =  Crewman  #1 

In  the  JCA,  because  the  pilot  and  co-pilot  have  redundant 
capabilities  dedicated  to  the  singular  purpose  of  flying  the 
aircraft,  the  person  was  mapped  to  the  Job  functionally. 

Crewmen  #1  and  #2  = 

Pilot/Co-Pilot  Airborne  through  Landing 
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Mission  Profiles 


Long  range  cargo 

Short  range  troop 

Low  altitude  air 
drop 

Takeoff 

Threats: 
Armor-Piercing 
Incendiary  (API) 
and  Man-Portable  Air 
Defense  Systems 
(MAN  PAD) 

ORCAJob: 
Airborne  — ►  Landing 

Threats: 

API'S  and  MANPAD'S 

ORCAJob: 
Airborne  — ►  Landing 

N/A 

Vignettes 

Cruise 

Threats: 
High-Explosive 
Incendiary  (HEI) 

ORCAJob: 
Airborne  — ^Landing 

Threats: 

HEI'S 

ORCAJob: 
Airborne  — ^Landing 

Threats: 

API'S  and 
MANPAD'S 

ORCAJob: 
Airborne  — ^Landing 

Landing 

Threats: 

API'S  and  MAN  PAD'S 

ORCAJob: 
Descent  — ^Landing 

Threats: 

API'S  and  MANPADs 

ORCAJob: 

Egress 

& 

Descent  -»Landing 

N/A 
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(▼}  RDECOMJ*  Personnel  Inputs  -  Target  Description  flfl 


•  The  ORCA  personnel  geometry  is  inserted  into  the  component-level  target 
geometry  and  the  ORCA  man  is  articulated  into  the  proper  crew  configuration. 


Body  armor  is  modeled  in  the  target  geometry. 

-  Pilot  and  Copilot  modeled  with  front  plates  only  and  Air  Warrior  helmets 

-  Loadmasters  and  troops  modeled  with  front  and  back  plates 


4  Crew  and  40  Troop  Configuration 


ORCA  man  with  front  plate  and  helmet 
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•  ORCA  utilizes  the  Abbreviated  Injury  Scale®  (AAAM,  Version  2005  Update  2008)(AIS).  AIS  is  an 
anatomically-based,  consensus-derived,  international  severity  scoring  system  that  classifies  each  injury  by 
body  region  according  to  its  relative  importance  on  a  6-point  ordinal  scale.  AIS  values  provide  information  on 
the  type,  location,  and  severity  of  anatomical  injuries.  AIS  scores  each  single  injury. 

•  MAIS  -  Maximum  Abbreviated  Injury  Score  (MAIS)  is  an  anatomical  measure  of  injury  severity.  This  score 
classifies  injury  severity  on  the  basis  of  the  single  injury  having  the  greatest  AIS  severity  value.  The  MAIS  is 
between  0  and  6. 

•  For  the  JCA  analysis,  if  any  crew  member  or  troop  received  a  serious  or  greater  injury  (MAIS  >  3)  then  the 
result  is  a  mission  abort. 

•  A  serious  Injury  -  An  injury  that  requires  immediate  medical  attention.  Serious  injuries  present  a  serious 
threat  to  life. 


MAIS 

Injury  Level 

Type  of  Injury 

1 

Minor 

Superficial 

2 

Moderate 

Reversible  injuries;  medical  attention  required 

3 

Serious 

Reversible  injuries;  hospitalization  required 

4 

Severe 

Life  threatening;  not  fully  recoverable  without  care 

5 

Critical 

Non-reversible  injuries;  not  fully  recoverable  even  with  care 

6 

Maximal 

Nearly  Unsurvivable 
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RDFCOMJ} 


Personnel  Inputs  -  Incapacitation 
Operational  Casualty 


ORCA  provides  operational  metrics  to  determine  if  personnel  have  the 
capability  to  meet  the  requirements  of  the  job  under  evaluation.  There 
are  six  post-wounding  times:  immediate,  30  seconds,  5  minutes, 

1  hour,  24  hours,  and  72  hours. 

•  Operational  ..Casualty" 

-  Applied  to  each  crew  member  for  specific  discrete  times 

-  If  the  task  elements  of  all  the  tasks  are  greater  than  the  minimum 
required  performance  level,  then  casualty  equals  0  (able  to  perform  job). 

-  If  one  of  the  task  elements  is  less  than  the  minimum  performance 
requirement,  then  casualty  equals  1  (unable  to  perform  job). 

•  For  the  JCA  analysis,  3  failure  modes  were  evaluated: 

-  Loss  of  Pilot 

-  Loss  of  Co-pilot 

-  Loss  of  Pilot  and  Co-pilot 
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•  System-Level  Vulnerability  Analysis: 

-  Pilot  and  Copilot  are  assessed  as  any  other  flight  critical  subsystem 

-  Pilot  and  Copilot  are  modeled  as  being  multiply  vulnerable;  either  crew  member 
can  fly  and  land  aircraft 

•  Three  System  Kill  Levels: 

-  Attrition:  Incapacitation  of  both  Pilot  and  Copilot  in  less  than  30  seconds 

>  Assess  the  probability  of  pilot  and  copilot  being  capable  of  performing  job 
>Use  ORCA  Operational  Casualty  metric 

>  Determine  if  pilot  and  copilot  fall  below  the  minimum  performance  level 

>  Assign  crew  member  a  1  (can  perform  job)  or  a  0  (can  not  perform  job) 

-  Fly  and  Land:  Inability  to  fly  and  land 

>  Flying  time  is  vignette  specific  with  time  periods  of  5,  15  or  30  minutes 

>  Use  ORCA  Operational  Casualty  metric 

-  Mission  Abort:  Any  crew  or  troop  member  receives  a  MAIS  >  3 

>  A  MAIS  >  3  is  a  serious  injury  that  requires  immediate  medical  attention 

>  Attrition  supersedes  mission  abort  if  both  Pilot  and  Copilot  are  incapacitated 
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•  Analysis  data  was  used  to  support  the  JCA"s  Full  Rate 
Production  Milestone  Decision. 

•Higher-resolution  survivability  assessments  are  available 
for  measuring  the  value  of  improved  air  systems. 

•  ORCA  is  an  invaluable  toolset  in  answering  the  questions 
of  how  the  individual  person  is  directly  tied  into  the  overall 
air  system. 
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Contact  Information 


Penny  Willard  41 0-278-7233 

pennv.willard@arl.armv.mil 

Richard  Moyers  410-278-4761 

richard.movers@arl.armv.mil 

U.S.  Army  Research  Laboratory 
Warfighter  Survivability  Branch  (WSB) 
Attn:  RDRL-SLB-W 
328  Hopkins  Road 

Aberdeen  Proving  Ground,  MD  21005 
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What  is  JTAPIC? 


The  Joint  Trauma  Analysis  and  Prevention  of  Injury  in  Combat  (JTAPIC) 
program  links  the  medical,  intelligence,  operational,  and  materiel 
communities  in  collecting,  analyzing,  and  integrating  data  from  combat 
incidents  to  inform  decisions  by  materiel  developers,  commanders, 
Training  and  Doctrine  Command  (TRADOC),  and  senior  leaders  to 
improve  Warfighter  survivability. 
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Recorded  Fragment  Information 


Mass:  2.25  g 

Dimension:  12.7  x  10.3  x  3.2  mm 
Density:  7.11  g/mL 
Shape:  Irregular 

Recovery  Location:  Neck  (pharynx) 


Description:  Smooth  copper  color.  Top 
concave  side  has  cut  marks.  Appears  to  be 
a  fiber-like  material  attached  to  fragment. 
Specimen  cut  for  analysis. 

Predominant  Materials:  Copper  and  Iron 


Photo  3-D  Scan 
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v  BDECOM  v  SEM-EDS  Fragment  Characterization 


Scanning  Electron  Microscopy-  Energy  Dispersive  X-ray  Spectroscopy  (SEM-EDS)  is 

an  analytical  technique  used  to  determine  the  elemental  composition  of  a  given  sample. 

•  Elemental  results  are  specific  to  the  nature  of  the  sample  and  the  surface  area  scanned. 

•  EDS  provides  a  first  approach,  qualitative  assessment  of  fragment  material. 


SEM  Image 


EDS  Spectrum 
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ICP-AES  Fragment  Characterization 


Inductively  Couple  Plasma- Atomic 
Emission  Spectroscopy  (ICP-AES)  is  a 

quantitative,  analytical  technique  used  to 
determine  the  elemental  composition  of  a 
given  sample. 

•  Metals  in  trace  amounts  can  be  detected. 

•  Exact  elemental  concentrations  and  metal 
alloys  can  be  determined. 


ICP-AES  Sample  Results 


Case  ID  # 

Chemical  Element 

Percent  of  Total  ( % ) 

Carbon 

0.46 

Sulfur 

0.024 

Manganese 

0.54 

Silicon 

.025 

Chromium 

0.39 

Nickel 

0.10 

Phosphorus 

0.009 

Copper 

0.16 

Molybdenum 

0.01 

Cobalt 

0.006 

Aluminum 

<0.002 

Lead 

<0.001 

Vanadium 

0.002 

Iron 

98.049 
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Database  Reporting 


Fragment  Brief  Report 

Generated:  2010-01-21, 122  cases,  822  fragments 


For  Official  Use  Only 

JTAPIC  Database 

Fragment  Body  Location  Report 
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mm 


TECHNOLOGY  DRIVEN.  WARFIGHTER  FOCUSED. 


Approved  for  Public  Release  -  Distribution  Unlimited 


8 


Approved  for  Public  Release  -  Distribution  Unlimited 


US  ARMY 


0  rdecom^  Incorporation  of  Fragment  Analyses 


Evidence 


Medical  Data 
and  PPE 
Description 

(OAFME) 


Operational 

and 

Intelligence 
Info  (NGIC) 


I 


i 


Benefits 


Armor  and  PPE 
Enhancements 


Improved  Methodology  for 
Survivability  Analyses 

TTP  Insights 

TECHNOLOGY  DRIVEN.  WARFIGHTER  FOCUSED. 


Approved  for  Public  Release  -  Distribution  Unlimited 


9 


US  ARMY 


RDECOM 


Approved  for  Public  Release  -  Distribution  Unlimited 

Example  of  Event  Recreation 
Using  Modeling  and  Simulation 


ARL  combines  fragment  information  with  the  operational  intelligence  and 
medical  data  received  from  the  other  JTAPIC  partners  to  analyze  and  recreate 
events  of  interest  using  modeling  and  simulation  (M&S). 
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New  Capabilities 


•  Analyzing  plastic  fragments  to  identify  plasticizer  and  polymer 
compounds 

•  Providing  support  to  Health  Affairs  for  embedded,  toxic  fragments 

•  Matching  fragments  to  anatomical  hit  locations 

•  Identifying  fragment  source  to  assist  the  material  development 
community  in  identifying  threat  materials 

•  Analyzing  fragment  masses  to  assist  the  material  development 
community  with  threat  identification  and  testing  designs 
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Future  Efforts 


•  Match  hit  locations  to  body  armor  placement 

•  Improve  visualizations  in  database  to  replicate  existing  prototypes 

•  Develop  a  classified  database  in  order  to  incorporate  operational 
intelligence  information,  medical  data,  and  fragment  analysis  results 
in  one  location 
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Karen  Pizzolato 

U.S.  Army  Research  Laboratory 

Temporary  Assignment,  Armed  Forces  Medical  Examiner  System 
Phone:410-278-4102 

Email:  karen.pizzolato@us.army.mil  or  karen.pizzolato@us.af.mil 
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What  is  JTAPIC? 


The  Joint  Trauma  Analysis  and  Prevention  of  Injury  in  Combat  (JTAPIC) 
program  links  the  medical,  intelligence,  operational,  and  materiel 
communities  in  collecting,  analyzing,  and  integrating  data  from  combat 
incidents  to  inform  decisions  by  materiel  developers,  commanders, 
Training  and  Doctrine  Command  (TRADOC),  and  senior  leaders  to 
improve  Warfighter  survivability. 
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Recorded  Fragment  Information 


Mass:  2.25  g 

Dimension:  12.7  x  10.3  x  3.2  mm 
Density:  7.11  g/mL 
Shape:  Irregular 

Recovery  Location:  Neck  (pharynx) 


Description:  Smooth  copper  color.  Top 
concave  side  has  cut  marks.  Appears  to  be 
a  fiber-like  material  attached  to  fragment. 
Specimen  cut  for  analysis. 

Predominant  Materials:  Copper  and  Iron 


Photo  3-D  Scan 
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SEM-EDS  Fragment  Characterization 


Scanning  Electron  Microscopy-  Energy  Dispersive  X-ray  Spectroscopy  (SEM-EDS)  is 

an  analytical  technique  used  to  determine  the  elemental  composition  of  a  given  sample. 

•  Elemental  results  are  specific  to  the  nature  of  the  sample  and  the  surface  area  scanned. 

•  EDS  provides  a  first  approach,  qualitative  assessment  of  fragment  material. 


SEM  Image 


EDS  Spectrum 
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ICP-AES  Fragment  Characterization 


Inductively  Couple  Plasma-  Atomic 
Emission  Spectroscopy  (ICP-AES)  is  a 

quantitative,  analytical  technique  used  to 
determine  the  elemental  composition  of  a 
given  sample. 

•  Metals  in  trace  amounts  can  be  detected. 

•  Exact  elemental  concentrations  and  metal 
alloys  can  be  determined. 


ICP-AES  Sample  Results 


Case  ID  # 

Chemical  Element 

Percent  of  Total  (%) 

Carbon 

0.46 

Sulfur 

0.024 

Manganese 

0.54 

Silicon 

.025 

Chromium 

0.39 

Nickel 

0.10 

Phosphorus 

0.009 

Copper 

0.16 

Molybdenum 

0.01 

Cobalt 

0.006 

Aluminum 

<0.002 

Lead 

<0.001 

Vanadium 

0.002 

Iron 

98.049 

Photograph  Courtesy  of  Image  from  Lehigh  Testing  Laboratories,  Inc. 

(www.lehightesting.com),  used  without  permission 
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Fragment  Detail  Information 
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Example  of  Event  Recreation 
Using  Modeling  and  Simulation 


ARL  combines  fragment  information  with  the  operational  intelligence  and 
medical  data  received  from  the  other  JTAPIC  partners  to  analyze  and  recreate 
events  of  interest  using  modeling  and  simulation  (M&S). 
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New  Capabilities 


•  Analyzing  plastic  fragments  to  identify  plasticizer  and  polymer 
compounds 

•  Providing  support  to  Health  Affairs  for  embedded,  toxic  fragments 

•  Matching  fragments  to  anatomical  hit  locations 

•  Identifying  fragment  source  to  assist  the  material  development 
community  in  identifying  threat  materials 

•  Analyzing  fragment  masses  to  assist  the  material  development 
community  with  threat  identification  and  testing  designs 
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Future  Efforts 


•  Match  hit  locations  to  body  armor  placement 

•  Improve  visualizations  in  database  to  replicate  existing  prototypes 

•  Develop  a  classified  database  in  order  to  incorporate  operational 
intelligence  information,  medical  data,  and  fragment  analysis  results 
in  one  location 
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Karen  Pizzolato 

U.S.  Army  Research  Laboratory 

Temporary  Assignment,  Armed  Forces  Medical  Examiner  System 
Phone:410-278-4102 

Email:  karen.pizzolato@us.army.mil  or  karen.pizzolato@us.af.mil 
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Autonomous  Systems 
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Autonomous  Systems  Capability  Leader 

trov@draoer.com 

(617)  258-2635 

Team  Members 

George  Sass,  Melissa  Durfee,  Nick  Borer,  Stephen  York,  Eric 
Nelson,  Mike  Ricard,  Scott  Ingleton 
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Motivation 


•  Across  DoD,  lack  of  common  vision  for  how  to  assess 
performance  of  decision-making  systems 

■  Need  to  meet  needs  of  commanders,  acquisition,  and 
warfighter  communities  who  need  to  trust  system 
performance  when  needed,  safely 

■  Low  confidence  of  performance  in  difficult  conditions 

■  Intractable  to  physically  test  every  possible  condition 
■  Interesting  Anecdotes 

■  All  deployed  ground  robots  are  tele-operated 

■  Original  iRobot  Packbot  had  many  autonomous  driving 
features  -  they  were  removed 

■  US  Army  tends  to  use  automated  Takeoff/Landing  features 
of  Predators,  Air  Force  does  not 
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Project  Vision 


Apply  Draper  experience  in  System 
Engineering,  M&S,  Reliability  Analysis 

■  Investigate  use  of  Markov  Reliability 
Analysis  and  DOE  for  System-Level 
test  planning 

■  Complementary  with  increasing 
emphasis  on  Model-Based  design 
within  DoD 

■  Approach  similar  to  human 
performance  evaluation:  Inject  failure 
conditions  during  training  to  force  off- 
nominal  decisions 

■  Feedback  performance  data  to 
model  over  time  to  improve 
predictions  of  future  reliability  - 
continuous  improvement 

Selected  Unmanned  Underwater  Vehicle 
(UUV)  for  Case  Study 

■  Highly  autonomous  operations  in 
complex  environment 

■  Strong  interest  from  community  in 
testing  improvements 


Concept  of 
Operations 


Architecture 
Requirements 

Design 


Operations  & 
Maintenance 
System 
Verification  & 
Validation 


Integration,  Test 
&  Verification 


Implementation 


Concept  of 
Operations 

Architecture 

Requirements 


Operations  & 
Maintenance 
System 
Verification  & 
Validation 


Design 


Integration,  Test 
&  Verification 


Implementation 
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Testing  Robustness  to  Build  Confidence 


Increase  Test  Coverage  with  Failure  &  Environmental  Conditions 


Tests  Defined  Only 
by  Requirements 


Increased  Coverage 
by  Failure  & 
Expanded 
Environmental  test 
design 
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Behavioral  Markov  Reliability  Analysis 


Failure  Failure  Failure 
rate  =  a  rate  =  b  rate  =  c 


Operational  State 
System  Loss  State 


0  FL  1  FL 


•  System  Markov  Model 

■  System  component  connections  &  logical  dependencies 

■  Reliability  values  for  each  system  component  (MTBF) 

•  Model  Outputs 

■  Probabilities 

Any  failure  condition  over  system  life 
System  Loss 

■  Reliability  Metrics 

Overall  Reliability  (not  directly  used  in  this  project) 

Sensitivity  of  Overall  Reliability  to  failure  rates  of 
components  (used  to  rank  importance  of  failure  modes) 

•  Draper  developed  PARADyM  Tool 


dPx 

dt 

dP2 

dt 

dPj 

dt 

dP4 

dt 


— (ci  +  b  +  c  ^)P\  (t ) 


aPx(t) 

bP\{t) 


~(a+b+c)t 

P2(t)=ae-{a+b+c)' 
P2(t)  =  be~{a+b+c)' 


PA(t)  =  ce-(a+b+c)' 


P(System  Loss)  =  ^(System  Loss  States) 
Reliability  =  ^(Operational  States) 
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Process  Summary 


Required  Inputs 

■  Behavioral  Markov  Model 

■  Extreme  types  and  ranges  of 
environmental  conditions 

Simulation  Test  Design 

■  Perform  Markov  reliability 
sensitivity  analysis 

■  DOE  for  environmental  conditions 

■  Repeat  all  (or  top  subset)  failure 
conditions  for  each  experiment 

Simulation  Execution  &  Analysis 

■  Parallel  execution  of  test  cases 

■  Analysis  of  Variance  to  find  Main  & 
Interaction  Effects 

■  Rank  significant  factors  according 
to  reliability  sensitivity 

Final  Results 

■  Possible  (not  yet  attempted)  to 
extract  confidence  intervals  for 
performance  over  bounds  of 
operation 

■  Highest  significance  subset  of 
recommended  tests  to  exercise  in 
field 


Connectivity 
Behavioral  Model 


Environmental 
Factors  &  Levels 


Markov  Reliability  Analysis 


Design  of  Experiments 


Integrate  Failure  Cases 
w/  Each  Experiment 


Simulation 
Test  Matrix  (i) 


High  Fidelity  Vehicle  Simulation 
with  Failure  Injection 


=  ♦  == 

Simulation  Test 
Results  (j) 


Main  Effects  &  Interaction 
Analysis  (DOE) 


Ranking  Against  Reliability 


Field  Test 
Matrix 


V' 


Performance 

Confidence 

Intervals 
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Case  Study:  Generic  UUV 


Based  on  NUWC  MARV  UUV 

■  V  Diameter,  12’  Long 

■  Max  Speed:  5  knots 

■  Prop  Driven  with  4  Control 
fins 

■  Forward,  Left,  Right,  Down 
Looking  Sonars 

ASTM  F41  Software 
Architecture 

■  Primary  decision  making  in 
Autonomous  Controller 
(AC) 

■  Vehicle  management  by 
Vehicle  Controller  (VC) 

■  Payload  operations 
through  Payload  Controller 
(PC) 

■  “Backseat  Driver” 

Paradigm  of  control 


AC 

Performs  Mission 


» 


v 

PC 

Controls 

Vehicle 

Payload 

Sensors 


* 


Planning,  Commands 
Steering  &  Speed, 
Payload  Use 
Scheduling,  SA 


V 


VC 

Performs  Low- 
Level  Vehicle 
Control  and 
Management 
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UUV  Simulation  Based  Testing 


New  Failure  Nodes  Inserted 


Draper  Simulation  Framework  (DSF) 

■  Govt.  Open  Framework 

■  Dynamics/Physics  simulation 

■  Soft  to  Hard  Real-Time  and  faster 

■  Built  for  Hardware-in-Loop 
MARV  UUV  Simulation 

■  Validated  vehicle  dynamics 

■  Simplified  sensor  models 

■  Autonomy  Controller  running 
Software-in-Loop  with  simulated 
environment 

New  Extensions  to  Simulation 

■  Created  generalized  failure  injection 
nodes  for  DSF 

■  Failure  types:  Omission/Constant, 
Noise,  Bias 

■  Parallel  execution  of  simulations  & 
Autonomy  Controllers 
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UUV  System  Responses 


Response 

Description 

Rationale 

Position  Error  (t) 

Deviation  from  baseline 
mission  path  over  time 

Position  errors  cause  data 
collection  errors 

Attitude  Error  (t) 

[<p,9,hj] 

Deviation  from  baseline 
attitude  over  time 

Attitude  errors  cause  data 
collection  errors 

Speed  Error  (t) 

Deviation  from  baseline  speed 
over  time 

Speed  influences 
execution  time,  stealth, 
energy 

Energy  Consumption 

Energy  consumption  for 
mission 

Must  operate  within 
available  energy  limits 

Mission  Time 

Total  mission  time 

Establish  expectations  for 
recovery/communication 

Surface  Position  Error 

Deviation  from  designated 
end-of-mission  surface  point 

Large  errors  on  surfacing 
impact  recovery 

Vehicle  Recoverable 

TRUE  if  vehicle  surfaced 

Lost  at  sea? 
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Case  Study  Evaluation  Scenario 


Scenario  Goals 

■  Short,  rapid  to  iterate 

■  Exercises  terrain 
avoidance 

■  Exercises  waypoint 
following 

■  Varies  ocean  currents, 
map  quality 

Case  Study  Scenario  Design 

■  Short  mission,  -  300 
seconds 

■  Approach  &  avoid  terrain 
on  way  to  waypoint 

Basis  of  all  case  study 

simulations 

Future  Scenario  Designs 

■  Longer  missions 

■  More  terrain  complexity 

■  Multiple  time-varying 
objects  of  interest  (ships, 
mines) 
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Environmental  Experiment  Design 


•  Available  Environmental  Factors  (3) 

■  Uniform  current  magnitude  & 
direction 

■  Terrain  under  vehicle 

•  DOE  Design 

■  2  Level,  3  Factor  Full  Factorial  - 
using  min/max  levels,  but  adding 
median  center  point  experiments 

■  Center  points  show  non-linearity  in 
response,  inform  analysis 


Min 

Median 

Max* 

Current 

Magnitude 

0  Knots 

2  Knots 

4  Knots 

Current 

Direction 

0° 

CO 

o 

o 

180° 

Map 

Mismatch 

0% 

50% 

100% 

Experiment  Design  with  Center  Points 
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Actual  Terrain  A  priori  Terrain  Map 


3/8/11  -  Learned  4knot  Odeg  current  cases  too  strong  for  vehicle 
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Example  Results:  Position  Response 


UUV  Path  During  Select  Bathymetric  Sonar  Failures 


Nominal  Failure  Case 

Bathy  Fails,  Map  Mismatch  100% 

Bathy  Fails,  50%  Map  Mismatch,  2  knot  Side  Current 
Bathy  Fails,  4  knot  Tail  Current,  Mission  Incomplete 
System  Baseline 


300 


Downrange  (ft) 


o  -400 


300 


Crosstrack  (ft) 


UNCLASSIFIED  PUBLIC  RELEASE 

Copyright  2011  by  the  Charles  Stark  Draper  Laboratory,  Inc.  all  rights  reserved 


March  16,  2011 


Example  Results:  Map  Mismatch  Effects 


•  Map  Mismatch  Significant 
Influence  during  Sonar 
Failures 

■  Logical  result 

■  Almost  4  km  Max 
Error  in  Surface 
Position 

■  From  Markov 
model,  sonar 
failures  drive 
reliability 

■  Fin  &  attitude 
sensor  failures 
much  less  probable 

■  Failure  effects  same 
magnitude  as 
environment  only 

■  Suspect  impact 
cases  and  4knot 
head  currents 
biasing  results 

•  Need  to  set  bounds  on 
responses 

■  Define  overall 
PASS/FAIL  limits 

■  Summarize  high 
level  results  more 
clearly 


Boxplot  of  Surface  Position  Error 
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Summary  &  Future  Work 


•  Demonstrated  Reliability  +  DOE  Test  Planning  method  on  Generic  UUV  case 

■  Reliability  analysis  indicated  sonars,  battery  monitor,  VC,  and  AC  primary  drivers 
of  system  reliability 

■  DOE  Planning  and  analysis  indicated  Map  Mismatch,  Current,  subset  of  failure 
modes  significant 

•  Need  to  complete  analysis  of  simulated  experiments 

■  Review  results  with  engineering,  end-users,  and  customers  to  get  feedback  on 
usefulness 

■  Rank  effects  and  interactions  against  probability  of  failure  conditions 

•  Invest  in  method  &  tool  improvements 

■  Simulation  Environment:  Needs  more  fidelity  in  water  properties,  coupled  with 
higher  fidelity  sensor  models 

■  Simulation  Environment:  Integrate  reliability  calculations  with  dynamic  system 
model  ->  Avoid  second  model  creation  effort 

■  Markov  Analysis:  Sources  of  reliability  values  (MTBF)  for  each  component 

■  Simulation  Environment:  Add  failure  mechanisms  for  VC  and  AC  during 
simulation 

■  Simulation  Environment:  Integrate  autonomous  controller  decision  logs  with 
response  data 

■  Simulation  Environment:  Add  time-varying  failure  and  environmental 
perturbations  during  simulation 

■  Design  of  Experiments:  Also  consider  for  integration  with  Simulation 

■  Design  of  Experiments:  Selection  of  best  designs  and  analysis  strategies  for 
higher-order  experiments 
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Ongoing  Testing  Efforts  of  Note 


•  Assistant  Secretary  of  the  Navy  (ASN)  Research  Development  and  Acquisition  (RDA) 

■  Large  scale  multi-unit  test  scenarios  with  many  interoperating  systems 

■  Amy  Markowich 

•  Marine  Corps  Warfighting  Lab 

■  Extensive  hands-on  evaluation  of  aerial/ground  robotics  in  relevant  environments 
&  missions 

■  Jim  Lasswell 

•  NAVSEA  (Combatant  Craft  Division) 

■  In-Water  testing  of  USV,  advocates  for  division  of  testing  at  key  interfaces  - 
Perception,  Effectors,  Planning  &  Control 

■  Eric  Hansen 

•  US  Army  Maneuver  Battle  Lab 

■  Live/Virtual/Constructive  testing  with  manned  and  unmanned  systems 

■  Harry  Lubin 

•  Army  Research  Laboratory  (ARL) 

■  Autonomous  ground  vehicle  behavior  testing  with  NIST  partnership 

■  Marshal  Childers 

•  MIT  PATFrame 

■  TRMC  funded  development  of  test  planning  framework  for  SoS 

■  Ricardo  Valerdi 
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Example  Results:  Current  Direction  Effects 


•  Current  Direction  Strong 
Effect 

■  Logical  result 

■  Almost  4  km  Max 
Error  in  Surface 
Position 

■  From  Markov 
model,  sonar 
failures  drive 
reliability 

■  Fin,  Prop,  &  attitude 
sensor  failures 
much  less  probable 

■  Failure  effects  same 
magnitude  as 
environment 

■  Suspect  impact 
cases  and  4knot 
head  currents 
biasing  results 

•  Need  to  set  bounds  on 
responses 

■  Define  overall 
PASS/FAIL  limits 

■  Summarize  high 
level  results  more 
clearly 
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UUV  Path  For  Select  Forward-Looking  Sonar  Failures 


Nominal  FLS  Failure  Case 
FLS  Fails,  Map  Mismatch  100% 

FLS  Fails,  Map  Mismatch  50%,  2  knot  Side  Current 
FLS  Fails,  4  knot  Tail  Current,  Mission  Incomplete 
System  Baseline 


300 


0  -200 


Downrange  (ft) 


Crosstrack  (ft) 
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Using  Complementary  Frameworks 
for  Qualitative  Data  Collection  during 
OT&E:  Piggybacking  on  Operational 
Experiments 


Chiesha  Stevens,  M.S. 
Nancy  Heacox,  Ph.D. 


Pacific  Science  &  Engineering  Group 

www.pacific-science.com 

9180  Brown  Deer  Road 
San  Diego,  CA  92121 
858-535-1661 


Integrated  Defense  Acquisition,  Technology,  and 
Logistics  Life  Cycle  Management  System 


Following  the  Materiel  Development  Decision,  the  Milestone  Decision  Authority  may  authorize  entry  into  the  acquisition  process  at  any  point,  consistent  with  phase-specific  entrance  criteria  and  statutory  requirements 


Operations  &  Support  Phase 


Materiel  Solution  Analysis  Phase 


Technology  Development  Phase 


Engineering  &  Manufacturing  Development  Phase 


&  Deployment  Phase 
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Tyeirsof 

[Funds 


Planning, 
Programming, 
Budgeting 
&  Execution 
Process 

(biennial- 

calendar-drivon) 
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Low-Rate  Initial  Production  Process 


Low-Rate  Initial 
Production 
Systems 


Final 

Product 

Baseline 


Product  Support  Package/PBL  Implementation 

•Product  Support  Elements  ‘Supply  Chain  Management 

•Support  and  Cost  Baseline 

•Contract  for  Sustainment  (organic  and  commercial) 


Independent  IOT&E 


Full-Up  System  Level  LFT&E 

Joint  Interoperability 
Certification  Testing 

JTIC  Joint  Interoperability 
Test  Certification 

INPUT 


OUTPUTS 


•Test  Results 

•Product  Baseline 

•Service 

•Exit  Criteria 

•Test  Reports 

•User  Fe 

•APB  •CPD  *SEP  •TEMP 

•SEP  •PESHE  -TEMP 

•Discref. 

•Product  Support  Element 

•System  Safety  Analysis 

•SEP 

Requirements 

•Inputs  to:  -IBR 

•System 

•PESHE 

-  Cost/Manpower  Est. 

•Produc 

•System  Safety  Analysis 

-  Product  Support  Package 

Requit 

Analyze  Deficiencies 
To  Determine  Corrective 
Actions 


Verification/ 
. Validation . 


PCA 

Verify  and  Validate 

. 3>- 

Production 

Configuration 

Modify  Configuration 
(Hardware/Software/Specs) 
To  Correct  Deficiencies 


Me 

Serv 

Chain 
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Operational  Test  and  Evaluation  (OT&E) 


Used  to  determine  the  operational  effectiveness  and 
suitability  of  a  system  under  realistic  operational 
conditions,  including  joint  combat  operations; 

•  determine  if  thresholds  in  the  approved  Capability 
Production  Document  and  critical  operational  issues  have 
been  satisfied; 

•  assess  impacts  to  combat  operations; 

•  and  provide  additional  information  on  the  system's 
operational  capabilities. 

Typical  users  shall  operate  and  maintain  the  system 
or  item  under  conditions  simulating  combat  stress  and 
peacetime  conditions.  i - 1 


Joint  Interoperability 
Certification  Testing 


JTIC  Joint  Interoperability 


Test  Certification 


The  DoD  OT&E  Agencies 


The  four  military  departments  have  each  formed 
operational  test  agencies  that  conduct  OT&E 
independently  of  the  acquiring  organizations. 

1 .  Army  Test  and  Evaluation  Command 
»  Operational  Test  Command  (OTC)  and 
»  Army  Evaluation  Center  (AEC) 

2.  Navy  Operational  T&E  Force  (OPTEVFOR) 

3.  Marine  Corps  Operational  T&E  Agency  (MCOTEA) 

4.  Air  Force  Operational  T&E  Command  (AFOTEC) 


Piggybacking  on  Operational  Experiments 


An  operational  experiment  may  be  a  suitable  venue 
and  afford  efficiency  of  the  IOT&E  testing  process 

-  Operational  experimentation  may  occur  as  an 
experimental  venue,  or  in  conjunction  with  an  operational 
exercise  already  being  planned 

Benefits: 

-  Temporary  installation  on  operational  platforms  -  ‘field  test’ 

-  Specifically-designed  scenarios  /  test  plans  -  ‘realistic 
combat  conditions’ 

-  Active-duty  participants  -  ‘use  by  typical  military  users’ 

-  Performance  measurement;  data  collection,  analysis,  and 
evaluation  -  ‘determining  effectiveness  and  suitability’ 


Operational  Experiments  Provide 
Administrative,  Logistics,  Data  Collection 

Administrative  processes  and  resources 

-  Test  design  and  planning  through  reporting 


Target  user  populations 

Official  entrance  to  operational  sites  and  platforms 

-  Access  to  networks  at  needed  classifications 

-  Support  for  installation 

-  Specific  needs  accommodated  as  feasible 


Test  plan  management  and  data  collection 
resources 

Independent,  objective  data  collectors 


An  Integrated  Data  Collection  Process 


Assessment  Plan 


-  Determine  the  system  or  technology’s  readiness  to 
participate  in  the  testing 

Test  Implementation 

-  Develop  work  process  models  for  use  of  the  systems 
or  technologies  in  mission-based  operational  events 

Results  Reports 

-  Provide  evaluations  as  to  the  operational  readiness  of 
each  system  or  technology 


Report  significance 
of  findings  based 
on  user  impact  and 
system 

performance  risk 
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Develop  a 
user-centered 
assessment 
plan  to  support 
IOT&E 


System  Capabilities 
Perform  ance 


Quality 


Compatibility 


Interoperability 


Participate  in 
IOT&E 


I 


Validate  HFE-based 
system  changes 

Collect  user  feedback  and 
human  performance  data 


A  Sampling  of  Venues 


Trident  Warrior 

-  Sponsored  by  U.S.  Fleet  Forces  Command 

-  Broad  spectrum  of  technologies;  multi-national  annual  focus 

Empire  Challenge 

-  Sponsored  by  Undersecretary  of  Defense  for  Intelligence 

-  ISR  processes  &  government-sponsored  technologies  with  minimum 
TRL  of  5  or  Milestone  B 

Talisman  Saber 

-  Sponsored  by  U.S.  Military  and  Australian  Defence  Force 

-  Technologies  &  processes  for  crisis  action  planning  and  contingency 
response 

Valiant  Shield 

-  Sponsored  by  U.S.  Pacific  Command 

-  Cooperative  detection,  tracking  &  engagement  of  units  at  sea,  in  air  and 
on  land 


....And  many  more.... 


Complementary  Data  Collection  &  Analysis 


Operational  experiments  require  -  or  can  accommodate 
the  strict  data  collection  requirements  of  IOT&E 

-  Specification  of  essential  system  attributes  to  be  tested 

»  Key  Performance  Parameters  (KPPs) 

»  Experimental  question,  with  attributes,  related  to  operational  capability 

-  Specification  of  measures 

»  Measures  of  Effectiveness  (MOEs)  and  Measures  of  Suitability  (MOSs) 
»  Criteria 

-  Specification  of  method 

»  Quantitative  or  qualitative 

»  Coordination  via  test  plan  (location  /  activity  /  timing) 

»  Multi-method  for  corroboration 

-  Specification  of  analysis 

»  Tests  /  Comparisons  to  be  performed 


Parallel  Activities:  Acquisition  Cycle 
&  the  Operational  Experiment  Cycle 


Needs/  —  What's  —  Performance  —  T&  E  Strategy  —  Laboratory  —  Data 
Gaps  oyl  there?1  Parameters  Testing  Plan 


Execution  /  —  Te&t  Results 
IOT&E 


Operational  —  Sponsor 
Assessment  Decision 


|  D  -  16  ma. 


0-12  mo. 

0-  IDrpo. 

L2; 


6  me 


0-4  mo. 


Conoepl 

Development 


isfttus  tfat 
smure  ntisjpmwjt 
wnm  military 

vteton  /  rtesrfa 


T  aeftnetogy 
Harvest 


m&jurch  arid 

select 

rechnofogies  :p 


LOi  ■  ■  -ic-ii 

Experimental 
Objectives  and 
Prc£tiS5  Action 
Maps 

Dafma  yDttfS  &ria 
prWimrnary 
measures; 
captor#  tfw 
foasMn a  /'ns  is'} 
end  iTflU'  r’fr?  be'] 
jamcess  in  design 
dtetftams, 


D&vel&u 
Experiment 
Design  ana 
DoTmu  EvOtlt 


Lay  otA  a  brood 
t&tfpte n  Qf*i 
develop 

sc&nm&s;  tioftiH} 
a  tetsHod 
BkentJlioo  plan 


Risk  Reduction 
1  Limited  Objective 
ExpannrarH  I'RR- 
I OE> 


Tanhitpfosy 
\nstallaiion  and 
wse  m  a 
simulated 
operational 
OflWtmirnw^  to 
assess  rfsA:  to 
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fnstafiatmn  site 
Site  or 

event  exmtmn 


0-2  mo. 


I  (OTTR3 

E^ec  ■ 

i  In  2 


Time  0 
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Daia  Collnclion 
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coffwfW  to  the 
means  -  ensure 
availability  of 
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noeted  -user 

group 


EMcutiem 

BfdoL/te  Jftil 
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rreed  Jo 
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chamog 
environment 


Prapara  Flflal 
Report 


'/diVdcift:.  data  and 
prows s  motets: 
expj'a^i  whethet 
tetfmeiBgy 
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experimental 
otijecti'/es 
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Op^alional 
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Experimental 

Assessment 


BetBirntpetton  by 
rindependenJ 

hanrdnf  utility/ 

potential  utility  to 
sponsoring 
cumrnaJiftsf 
offitnifitaikim 


Capa^il1id&- 

Based 

Assessment 


M rj'u ■  o  Davetoprrwnt  of 

_  Deoyalopment  _  Program  Slrstegrv 
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foL-l  Evaluation 
Strategy 
Development 


Day.  Test  &  Evai 
_  t'DT&Ej.  E-'arly 
OpRfatiariBi 

Assessment 
(EDAj  and 
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Assessment  [QAj 


' 


TteJ  and 
Evaluation 
MesIpt  Plan 
[TEMPI 
pavelopmenr 


Operational 

Tes* 

RrtitlluHfJS 

Review 

lOTRR,. 


Ihdependoril 


Tes*  and 

Evaluation 

<IQT*E} 


Production  l4 


Production  or 


Full-Rate 

^initial  Operational  Operational  Teal  _  Beyond  l  ow  Rate  M  Prnc jehon 

□edsan  Review 

.FpPDRj 


.Agency  Report  qf 

OISE  Results 


Initio,:  Production 
[BlRI  P>  Report 


Asessmeat  tn  ensnm 
that  the  prodtJctirm 
configuration  system 
C&npfQGQQd  \PtO 
tnyfiei  Operational 
Test  end  Evaluation 
(iGT&E)  with  a  high 
probability  of  success 
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Acquisition  Cycle  Source  Document 
Content  Used  for  Op  Ex  /  IOT&E  Test  Plan 


Concept  & 
strategy  for 
technology 
development 


Master 
schedule  for 
tests  &  other 
essential 
activities 


Threshold  & 
objectives  of 
user 

operational 

performance 


Initial 
.Capabilities. 
Doc 


Acquisition 

Strategy 


Capability 

Develop. 

Document 


TES  »>  TEMP  (living  doc) 


- M — £ - 

Analysis  of 

System 

- M - 

Acquisition 

;r  Alternatives, 

,  Threat  , 

Program 

Assessment 

Baseline 

_ _ 

.woer  mos  & 

MOP  to  ID 
elements  for 
oval  &  select 
tests 


Threat  &  threat 

environment: 

derive 

scenarios  from 
here 


KPPs  as 
drivers  of  Op 
Effectiveness 
&  Suitability 


Develop  a  user- 
centered 
assessment  plan 
to  support  IOT&E 


System  Capabilities 

Performance 

"Quality 


Compatibility 

Interoperability 


Participate  in 
IOT&E 


Collect  user 

feedback  and 
human  performance 
-data 

Validate  HFE-based 
system  changes 


Report  significance  of 
find!  ogs  based  on 
user  impact  and 
system  performance 
risk 
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Example  of  Qualitative  Templates  - 

Building  Blocks  for  Operational  Experimentation 
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Summary 


T&E  professionals  can  leverage  operational 
experiments  to  efficiently  fulfill  requirements  for 
IOT&E 

-  Operational  experiments  provide  a  management 
process  foundation  and  materiel  resources 

-  Acquisition  cycle  documents  and  reports  match  with 
needed  operational  cycle  documentation  -  minimal 
reworking  for  participation 

-  IOT&E  requirements  for  test  components  -  venue, 
scenario,  operators,  customized  and  objective  data 
collection  -  can  be  met 

-  Event  execution  can  be  overseen  by  operational  test 
agency  personnel  for  validation  to  the  Director  of 
Operational  Test  and  Evaluation 


For  more  information ,  please  contact: 

Pacific  Science  &  Engineering  Group 

Chiesha  Stevens 

ChieshaStevens@pacific-science.com 

(858)535-1661 
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•  USS  Cole 

•  October  12,  2000 

•  No  force  protection  equipment, 
plans,  or  training 

•  Killed  17  injured  39 

*  Anti-Pirate  patrols  in  Gulf  of 
Aden  and  Indian  Ocean 

•  Late  2007,  US  Navy  began 
stepping  up  anti-piracy  efforts 
when  received  permission  to 
enter  Somali  territorial  waters. 

•  Jan  2009,  the  US  Navy  in 
conjunction  with  20  other 
nations  formed  the 
international  anti-piracy  fleet, 
Task  Force  151. 
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*  Iran  posturing  in  the  Hormuz 
Strait 

•  Iranian  Navy  consists  primarily 
of  small  patrol  boats. 

•  Feb.  of  2007,  began  an 
increase  in  probing  of  Iraqi 
territorial  waters 

•  March  of  2007,  held  15  British 
Marines  and  Sailors  hostage 
for  a  short  time 

•  January  2008,  five  Iranian 
patrol  boats  took  aggressive 
action  and  “maneuvered  within 
500  yards  of  our  ships” 
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•  These  missions  require  tracking 
and  engagement  of  relatively  small 
boats. 

•  The  distances  to  the  vessels  are 
typically  short  range. 

•  The  primary  weapons  employed  are 
crew-served  weapons. 

•  Placing  sailors  on  the  gunwales 
with  crew-served  weapons  to 
engage  a  small  craft  bearing 
automatic  weapons  requires 
protection 
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*  Desert  Shield/Storm 

-  Ballistic  shields  were  installed  on  selected 
ships  at  the  crew  served  weapons  stations 
while  serving  in  the  Persian  Gulf  in 
support  of  Operation  Desert  Shield/Storm. 

-  Simple  laminated  Kevlar  panels. 

-  Represented  current  technology  at  the 
time 

*  Return  to  the  Gulf 

-  In  2003  CGs  and  DDG  received  shields  for 
operations  in  the  Gulf. 

-  Initially,  Desert  Shield/Storm  armor 
brought  out  of  storage  and  reissues. 

-  Some  new  design,  but  no  development 
with  respect  to  environment,  installation 
constrains,  or  even  threat  level  completed. 
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*  Degradation  and  replacement 
efforts 

-  Feb  2007,  SEA06  AT/FP  TWH  email 
to  OPNAV,  FFC,  CNSF  urging  shield 
resolution  (i.e.  life  cycle  support) 

-  2008  USS  Barry  realizes  a  need  for 
replacement  of  degraded  shields 
and  sources  own  shield. 

-  New  shields  not  authorized,  but  life 
cycle  support  not  in  place  for 
replacement  or  upgrade. 

-  Dec  2008  -  CNSF  sends  Crew 
Served  Weapon  Mount  Ballistic 
Shield  Requirements  letter  to 
Deputy  CNO 


Harnessing  the  Power  of  Technology  for  the  Warfighter 


•  This  project  will  develop  the  requirements  document  and  subsequently 
the  performance  specification  that  will  be  used  to  purchase  shipboard 
ballistic  shields. 

•  This  project  will  improve  the  ability  of  all  Navy  combatant  surface  ships 
to  meet  AT/FP  threats  through  the  use  of  ballistic  shields  that  meet 
requirements. 

•  Improved  ballistic  shields  will  reduce  the  risk  of  loss  of  life.  Current 
ballistic  shields  insufficiently  protect  ship'&  personnel  and  equipment 
against  documented  fleet  requirement.  Loss  of  life  safety  risk  exists 
with  currently  fielded  ballistic  shields. 

•  Standardization  of  ballistic  shield  requirements  is  expected  to  reduce 
overall  fleet  lifecycle  cost. 

•  Performance  spec  will  lead  to  a  common  ballistic  shield  product.  There 
is  currently  no  ballistic  shield  commonality  across  ship  classes. 

•  Formalized  performance  specs  will  allow  industry  the  ability  to  develop 
innovate  and  off  the  shelf  solutions. 
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Two  document  approach. 

-  MIL-PRF  document  identifying 
issues  unique  to  the 
installation  and  usage  of  the 
ballistic  shields  on  naval 
vessels. 

-  MIL-STD  document  addressing 
the  majority  of  possible  threat 
rounds  both  NATO  and 
WARSPACT.  It  will  provide 
comprehensive  testing, 
qualification,  and  classification 
standards  adaptable  to  all 
future  Naval  Ballistic 
Protection  needs. 
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•  DorT'tlimit  innovation 

-  Does  not  specify  materials 

-  Does  not  specify  mounting  methodology 

•  Encourage  all  solutions 

-  Covers  special  considerations  for 
permanent,  semi-permanent,  and 
removable  designs. 
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•  Provisions  unique  to  stationary 
and  removable  shields. 

-  Stationary  Shields 

•  Sea  State  Survivability 

•  RF  Signature 

•  RF  Reflectivity 

-  Removable  Shields 

•  Two  Man  Portable  (Weight,  etc.) 

•  Portability  Provisions  (Handles,  etc) 

•  Ease  of  Installation  (Markings,  Time 
to  assemble,  special  tools,  etc.) 

•  Passage  Way  and  Hatch  compatible 
(Dimensional  limits,  etc.) 
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Document  all  considerations 
and  constraints 


-  Includes 

•  Material  Handling 

•  Coatings 

•  Environmental  Testing 

•  Ship  Unique  Issues  (Green 
water  loading,  vibrations,  etc.) 

•  Flight  Operations 

•  Storage  Provisions 

•  Ship'&  Operations 
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•  Open  to  all  ship  classes 

-  Dimensions  are 
determined  by  the  lateral 
traversing  limits  based  on 
installation  and  the 
gunners  working  circle 
dimensions. 

-  Height  is  measured  based 
on  the  user 

•  48”  from  bottom  of  user"s 
feet. 

-  Weapon  cut  out  is 
determined  by  the  weapon 
mount. 


Maximum 
T raverse  position 


Not  to  Scale 


Required 

Lateral 

Zone 


Required 

Lateral 

Zone 


I  Perpendicular 
Projection 


i - 

:  2  inches  beyond 
Gunnersmate 
working  circle 


Gunnersmate 
working  circle 


Gunnersmate 
working  circle 


Maximum 
T raverse  position 


Not  to  Scale 


Required  Frontal  Zone 


:  2  inches  beyond 
|  Gunnersmate 
|  working  circle 
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•  NIJ  pros/cons 

-  Pros 

•  Excellent  and 
comprehensive 
procedures  for  body 
armor  applications 

-  Cons 

•  Limited  round  sizes;  not 
very  many  military 
rounds 

•  Ambiguous  multi-shot 
placement  criteria. 


Caliber 

Round 

Weapon 

NIJ  0101 06 

9x  19 

(9  mm;  .40  S&W) 

M9 

II A 

(9  mm;  .357  Magnum) 

Colt  Python 

II 

11  x  41 

(.357  SIG;  .44  Magnum) 

S&W  Model  2S 

IMA 

7.62  x  39 

Type  PS 

AK-47 

API  BZ  M43 

5.45  x  39 

5N7 

AK-74 

5.56  x  45 

M855 

M16 

7.62  x  51 

M80,  M59 

FN  FAL 

III 

AP  M61 

7.62  x  63 

M2 

Ml  Garand 

AP  M2 

IV 

7.62  x  54R 

SOVIET,  TYPE  LPS 

PKM 

Dragonuv 

Type  B32 

12.7  x  108 

12.7mm  API&T,  B32 

DShK 

12.7  x  99 

M2  Ball 

M2  BMG 

M2  AP 

14.5  x  114 

14.5mm  API-B32 

KPV 

14.5mm  API-BS-41 

20  x  102 

M75 

M61  Vulcan 

APT-M95 

AP-T  M602  (HVAP-T  DM-43) 

23  x  152 

23mm  API-T  BZT 

2A14 

25  x  137 

APDS-T  M791 

M242 

30mm 

30  x  113mm 

M230 

30  x  165mm 

GSh-30-1 

30  x  173mm 

GAU-8 
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•  EN  1063  pros/cons 

-  Pros 

•  Good  multi-shot 
placement 
methodology 

•  Included  military 
significant  rounds 

-  Cons 

•  No  Warsaw  Pact 
weapons 

•  Limited  threat  size. 


Caliber 

Round 

Weapon 

EN  1063 

9x  19 

(9  mm;  .40  S&W) 

M9 

EN  BR2 

(9  mm;  .357  Magnum) 

Colt  Python 

EN  BR3  i 

11  x  41 

(.357  SIG;  .44  Magnum) 

S  &  W  Model  29 

EN  BR4 

7.62  x  39 

Type  PS 

AK-47 

API  BZ  M43 

5.45  x  39 

5N7 

AK-74 

5.56  x  45 

M855 

M16 

EN  BR5 

7.62  x  51 

M80,  M59 

FN  FAL 

EN  BR6 

AP  M61 

EN  BR7 

7.62  x  63 

M2 

Ml  Garand 

AP  M2 

7.62  x  54R 

SOVIET,  TYPE  LPS 

PKM 

Dragonuv 

Type  B32 

12.7  x  108 

12.7mm  API&T,  B32 

DShK 

12.7  x  99 

M2  Ball 

M2  BMG 

M2  AP 

14.5  x  114 

14.5mm  API-B32 

KPV 

14.5mm  API-BS-41 

20  x  102 

M75 

M61  Vulcan 

APT-M95 

AP-T  M602  (HVAP-T  DM-43) 

23  x  152 

23mm  API-T  BZT 

2A14 

25  x  137 

APDS-T  M791 

M242 

30mm 

30  x  113mm 

M230 

30  x  165mm 

GSh-30-1 

30  x  173mm 

GAU-8 
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CRANE 


•  662  pros/cons 

-  Pros 

*  Excellent  for  categorizing  the  material  properties 
of  the  armor 

-  Cons 

*  Doesn"tgive  yes  or  no 

*  Allows  „gaming’bf  test  by  providing  for  obliquity 
and  offset  distance  from  muzzle 

*  Without  defined  levels,  difficult  to  develop  off  the 
shelf  materials 
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Harnessing  the  Power  of  Technology  for  the  Warfighter 


CRANE 


WARFARE  CENTERS 


•  Reviewed  the  majority  of  armor  related  standards  and  specs 

-  EN  1063 

-  NIJ  0101_06 

-  NIJ  0108_01 

-  MIL-STD-662F  V50  Ballistic  Test  for  Armor 

-  STANAG  4569 

-  MIL-DTL-46100E  Armor  Plate  Steel  Wrought  High  Hardness 

-  MIL-PRF-46103E  Armor  Lightweight  Composite 

-  MIL-PRF-46108C  Armor  Transparent 

-  ATPD  2352P  Transparent  Armor  Purchase  Specification 

-  MIL-B-29604(1)  Body  Armor  Hard  Small  Arms  Protective 

-  MIL-DTL-46063H  Armor  Plate  Aluminum  Alloy,  7039 

-  MIL-DTL-46077G  Armor  Plate  Titanium  Alloy  Weldable 
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Harnessing  the  Power  of  Technology  for  the  Warfighter 


•  Selected  best  practices  from  among  all 
reviewed  documents 

•  Massaged  given  info 

•  Filled  in  gaps  and  loopholes 

-  Current  Standards  primarily  NATO  rounds  only. 

-  Special  considerations  for  tiled  solutions 

-  No  obliquity  allowances 

-  Based  on  advertised  muzzle  velocity  of  given  threat 

-  Designed  to  easily  cross-reference  between  threat 
round,  common  weapons,  and  ballistic  properties. 
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Harnessing  the  Power  of  Technology  for  the  Warfighter 


CRANE 


•  Don"tlimit  innovation 

-  Does  not  specify  materials 

*  Encourages  new  chemical 
compositions  of  existing 
armor  materials. 

•  Encourage  all  solutions 

-  Allows  for  single  shot  or 
double 

-  Allows  for  ball  round  or 
armor  piercing 
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Harnessing  the  Power  of  Technology  for  the  Warfighter 


CRANE 


WARFARE  CENTERS 


•  Transparent  and 
opaque 

-  Allows  transparent  and 
opaque. 

-  Provides  small  changes 
based  on  typical  usage 

•  Thinner  witness  plate  for 
transparent 
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CRANE 


•  More  specific  shot 
placement 

-  Multiple  required 
locations  for  all 
coupons 

-  Special 
considerations  for 
tiled  coupons 


Adjacent 

Tiles 
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MIL-SI 

rD-X618 

Threat  Information 

r 

Existing  Standards 

i 

Type 

Class 

Caliber 

Round 

Weapon 

]NIJ  0101 06 

UL  752 

NATO  STANAG  4569 

EuroNorm  EN  1063] 

I 

A 

9x19 

9mm  FMJ  RN  M882 

M9 

1 

IIA 

1,6 

ENBR2  | 

B 

9mm  FMJ  RN 

Colt  Python 

n 

2 

ENBR3 

II 

A 

11x41 

.357  SIGFMJ  FN  A  A 19 

S  &  W  Model  29 

III  A 

3 

ENBR4 

III 

A 

7.62x39 

Type  PS 

AK-47 

B 

APIBZM43 

Level  2 

IV 

A 

5.45x39 

5N7 

AK-74 

B 

7N22  AP 

! 

! 

V 

A 

5.56x45 

M855 

M16 

! 

7 

Level  1 

ENBR5  j 

B 

APM993 

VI 

A 

7.62x63 

M2 

Ml  Garand 

! 

4 

B 

APM2 

1  IV 

9 

1 

VII 

A 

7.62x51 

M80,  M59 

FNFAL 

I  in 

5,8 

Level  1 

ENBR6  ! 

B 

APM61 

l 

Level  3 

ENBR7 

VIII 

A 

7.62  x54R 

SOVIET,  TYPELPS 

PKM 

B 

Type  B32 

Dragonuv 

1 

Level  3 

1 

IX 

B 

12.7  x  108 

12.7mm  API&T,  B32 

DShK 

X 

A 

12.7x99 

M33 

M2BMG 

10 

i 

B 

M263 

1 

XI 

A 

14.5x114 

14.5mm  API-B32 

KPV 

Level  4 

B 

14.5mm  API-BS-41 

1 

XII 

A 

20  x  102 

M75 

M61  Vulcan 

1 

| 

B 

APT-M95 

! 

XIII 

B 

23  x  152 

23mm  API-T  BZT 

2A14 

1 

i 

XIV 

B 

25  x  137 

APDS-T  M791 

M242 

1 

Level  5 

XV 

B 

30mm 

M789  HEDP 

M230 

1 

XVI 

B 

30mm 

30x1 65mm  BT 

GSh-30-1 

i 
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CRANE 


Wpi 

rf 

\  V  V  \ 

^  * 

*  MIL-PRF-XX613  and  MIL-STD-X618  are  in  the  Government 
Industry  Review  process. 

*  Both  documents  are  slated  to  be  signed  and  published  in 
late  March  2011. 

*  Following  the  signing  of  the  documents,  an  SBIR  will  be 
released  to  encourage  development  of  initial  designs. 

*  The  SBIR  will  bridge  the  gap  until  the  funding  request, 
currently  in  POM  cycle,  is  approved  allowing  shields  to  be 
fielded  on  DDGs,  FGs,  and  CGs. 
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CRANE 


*  NSWC  Crane  has  created  a 
Ballistic  Test  Group  to 
provide  the  required 
government  certification  for 
the  Navy. 

-  Ballistic  shots  up  to  and 
including  30mm 

-  Explosive  blasts  up  to 
500lbs 

*  EFPs  up  to  lOlbs 

*  A  50lbs  facility  is  being 
constructed. 
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•  Analysis  Background 

•  Modeling  Process  Overview 

— The  MUVES-S2  Model  for  Ballistic  Vulnerability  /  Lethality  (V/L)  Analysis 
— The  Model  Process 

— The  Importance  of  Highly  Detailed  Target  Geometry 

•  Target  Geometry  Development 

— System  Representation 
— Shot-Line  Sequence 
— Conversion  of  Vendor  CAD  Files 
— Building  High-Fidelity  CAD  Geometry 

•  Conclusions 
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Analysis  Background 


_  US  ARMY 

0  RDECOM 


•  All  vulnerability/lethality  efforts  follow  the  same  “general”  analysis  procedures. 

•  Inputs,  models,  and  methodologies  are  tailored  to  fit  particular  needs  of  the  customer: 

—  acquisition  decisions  (PMs  /  PEOs,  LFT&E  Community) 

—  system  design  /  armoring  initiatives  (PMs,  rapid  fielding  initiatives) 

—  personnel  survivability  studies  (PMs  /  PEOs) 

—  AoAs,  Army  Studies  (AMSAA,  TRADOC,  CAA) 

—  weaponeering  decisions  (JTCG) 

•  Fidelity  of  analysis  varies  from  a  high  level  of  detail,  as  in  component-level  analyses,  to  a  lower 
level  of  detail  as  dictated  by  customer  requirements. 

•  Benefits  of  modeling  and  simulation  (M&S)  to  the  LFT&E  community: 

—  Provides  a  “global”  interrogation  of  the  target,  utilizing  results  of  live-fire  events  to 
validate  MUVES-S2  M&S  results. 

—  Supplements  (not  substitutes)  the  LFT&E  process  with  a  more  global  interrogation  of  the 
vehicle. 

•  Results  are  highly  dependent  on  the  fidelity  of  the  inputs. 

—  Computer  aided  design  (CAD)  geometry  is  the  foundation  of  these  inputs. 
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Modeling  Process  Overview 
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The  MUVES-S2  Model  for 
Ballistic  Vulnerability/Lethality  Analysis 


Target  geometry 


Criticality  analysis 
of  subsystems 


Component 

vulnerability 


Personnel  data 


180° 


Fragmenting  Munitions 


Avg.  mass  groups 
Avg.  velocities 
No.  of  fragments 
Avg.  shape  factor 

Spray 
Zones 


Residual  penetration 
Personnel  incapacitation 
Component  damage 
Subsystem  capabilities 
Remaining  system  utility 
User-defined  criteria 
etc. 


Shaped  charge  Jet 

Kinetic  energy 
long-rod  penetrator 


Behind-armor  debris 
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The  Model  Process 
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A 
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The  Importance  of  Highly  Detailed 
Target  Geometry 


•  MUVES-S2  analyses  interrogate  the  target  utilizing  multiple  shot-lines. 
Examples  include: 

—  Artillery  rounds  create  multiple  shot-lines  that  generate  more 
opportunity  to  interact  with  subtle  details  of  the  geometry. 

—  Behind  armor  debris  evaluates  interior  components  of  the  vehicle 
as  the  threat  and  all  secondary  effects  interact  with  the  vehicle 
geometry. 


•  Accurate  geometry  is  essential  to  generate  quality  results. 
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Target  Geometry  Development 
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Abrams  Tank  on  Aberdeen 
Test  Center  Test  Pad 


Construct  3D  solid 
geometric  model 
of  system 
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High-Detail 

BRL-CAD™  Representation 
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Sample  Shot-Line  Sequence 


Glacis 

Armor 


Armor-piercing  HE 
Rounds  Round 


Fire 

Wall 


Engine  Starter  Transmission 

Sump 


Fan  Rear 
Armor 
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•  Native  CAD  files  provide  ARL/SLAD  with  higher  resolution  source  data  that 
facilitates  the  conversion  into  higher  resolution  BRL-CAD™  for  M&S  analyses. 
—  Vendor  CAD  files  are  preferred  method  of  geometry  development. 

—  ARL/SLAD  has  the  tools  to  receive  and  convert  multiple  formats  of  CAD. 


Vendor  Provided  Pro/E  CAD  Wireframe  Model  Final  BRL-CAD™  Rendering 


TECHNOLOGY  DRIVEN.  WARFIGHTER  FOCUSED^  2 


US  ARMY 


tyRDECOMjm  Conversion  of  Vendor  CAD  Files 


•A  highly  detailed  component  model,  comprised  of  multiple  solids,  is  reliant  on  a  thorough 
understanding  of  that  component’s  design. 


•  The  quality  of  the  resulting  BRL-CAD™  geometry  is  highly  dependent  on  the  quality  of  the  CAD 
that  is  provided. 


•Accurate  component  characteristics  to  include  dimensions,  thickness,  and  materials  is  desired 
and  achievable  with  CAD  files  that  include  more  detail  than  just  surfaces  (i.e.,  non-shrink 
wrapped  source  CAD). 


Vendor  Provided  Pro/E  CAD 


Wireframe  Model 


*More  detail  than  just  a  surface  model*  Final  BRL-CAD™  Rendering  .  0 
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Building  High-Fidelity 
CAD  Geometry 


•  Utilization  of  various  metrology  equipment  facilitates  a  high  degree  of  accuracy  in 
data  collection. 

—  This  process  requires  an  extended  period  of  time  with  the  vehicle  in  a 
“stable”  or  semi-controlled  environment. 

—  In  order  for  the  data  collection  process  to  be  efficient,  multiple  personnel 
with  various  pieces  of  equipment  are  required. 


Data  Collection  on  M1151A1  HMMWV  Raw  Data  Collected 

in  ARL/SLAD  Facility  In  Commercial  CAD  Software 
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Building  High-Fidelity 
CAD  Geometry 


•  While  not  the  preferred  form  of  geometry  development,  data  collection  through 
metrology  equipment  has  seen  advancements. 

—  Longer  lead-time  than  conversion,  but  faster  and  much  more  accurate  than 
older  “hand  measurement”  techniques. 

—  Facilitates  conversion  from  commercial  CAD  packages  to  BRL-CAD™ 
(necessary  format  for  MUVES-S2  simulation). 


M1151A1  HMMWV  Subsystems  Solids  Completed  M1151A1  HMMWV  Vehicle 

in  Commercial  CAD  Software  in  Commercial  CAD  Software 
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•  Modeling  and  simulation  can  supplement,  but  is  not  a  substitute  for,  live- 
fire  testing  to  provide  a  more  thorough  evaluation  of  vehicle  vulnerabilities 
and  armor  design. 

—  Provides  a  “global”  interrogation  of  the  target,  saving  assets 
(minimizing  cost)  as  well  as  maximizing  data  while  minimizing  the  test 
schedule. 

•Accurate  target  geometry  is  the  foundation  to  a  MUVES-S2  analysis. 

—  Accuracy  is  achieved  by  attaining  quality  vendor  CAD  geometry  to 
convert  into  BRL-CAD™. 

—  Adequate  time  on  a  representative  asset  is  required  to  facilitate  the 
necessary  vehicle  interrogation  for  geometry  development. 
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Purpose 


•  What  leads  to  a  successful  mission  effectiveness  assessment? 

•  How  do  you  set  yourself  up  for  a  successful  design  of  experiment? 

•  I’ll  describe  how  a  mission  decomposition  process  leads  to 

•  Successful  mission  effectiveness  assessments 

•  Improved  test  design  and  design  of  experiments 

•  Enhanced  mission-based  testing 

•  I’ll  provide  an  overview  of  a  structured  mission  decomposition  using 
the  Measures  Development  Standard  Operating  Procedure  (SOP) 
process  steps  as  an  example 
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Mission  Decomposition 

What  is  it? 

•  Methodology  for  understanding  the  contribution  of  a  system  under 
test  (SUT)  to  the  system-of-systems  (SoS),  task,  and  mission 

-  Enables  quantitative  measurement  of  system  and  task(s) 

-  Offers  the  ability  to  qualitatively  evaluate  the  mission 

•  Disciplined  and  repeatable  process  for  developing  relevant  mission, 
task,  and  system  measures 

-  Documented,  methodical,  and  thorough 

-  Therefore,  it  is  not  reliant  on  corporate  knowledge 

•  An  objective  mission-based  approach  to  designing  vignettes 

•  A  process  to  enhance  requirements  generation,  capability 
development,  and  testing 

Focused  on  the  ability  of  the  warfighter  to  perform  tasks 
and  achieve  mission  desired  effects 
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Mission  Decomposition 

So  What? 


Moves  the  focus  from  a  “systems  only”  approach  to  one  that 
deliberately  addresses  task  and  mission 

Enables  sufficient  conclusions  of  a  system’s  impact  on  combat  mission 
effectiveness 


-  Decomposes  a  warfighting  mission 

-  Traces  system,  task  ,  and  mission  relationships  to  warfighter  requirements 
Enhances  mission-based  testing 

-  Understanding  the  mission  and  task(s)  enables  better  understanding  of  the 
system  contribution  to  the  warfighter  and  the  mission 

-  Better  definition  of  test  priorities  (critical  vs.  ‘‘nice  to  have”  measures) 

-  Helps  confirm  that  an  identified  gap  has  been  successfully  addressed 

■  System-specific  attributes  alone  will  not  do  this 

Identifies  the  right  measures  to  answer  the  right  questions 

at  the  right  time 
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Mission  Decomposition 

An  Enabler  for  DOE 


Assists  with  defining  the  problem 

-  Based  on  a  capability  gap  derived  from  a  mission/task  analysis 

-  Mission,  effects,  capabilities 

Helps  determine  dependent  and  independent  variables 

-  Measures  of  mission  effectiveness  and  task  performance 

-  Conditions  of  the  environment,  threat,  and  joint 
Scopes  test  design  directly  to  the  SUT  capability  gap 

-  Leads  to  evaluating  warfighter  gap(s) 

-  Supports  scenario/vignette  selection 

-  Drives  data  requirements,  test  methods,  and  resource  requirements 
Places  focus  of  the  design  on  warfighter  requirements 


“Not  everything  that  can  be  counted  counts  and  not  everything 
that  counts  can  be  counted.”  -  Albert  Einstein 


Mission  Decomposition 

Major  Elements 

Does  your  evaluation  approach  provide  a  way  to 
determine  system  impact  on  task  and  mission  ? 


Mission 

Statement 

Objectives 

Desired 

Effects 


Tasks 

Sub-tasks 

Nodes 


•  Desired  Effects 
attributes  •  Attribute 

•  Task  attributes  measures  for 

•  SWarF  mission  and 

attributes  tasks 


Traceability  of 
system  attributes 
to  task  and 
mission  desired 
effects 


LU 

Rl  1 

mm  m 

i  — 

i 

Mission  . 
Description  1 

D> 

|  Des< 

rask 

cription  1 

Mission 

&  Task  - 
Attributes  1 

|  Measu 

Traceability: 
II  System-»Task  , 
res  ||  Mission  1 

r 
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Mission  Description 


( Describe  the  mission  in  terms  of  objectives  and  desired  effects  (outcomes). 

•  Identify  the  Mission  Statements),  Objectives  and  Desired  Effects 
from  authoritative  sources: 

-  JCIDS  documents  (ICD,  CDD,  CPD) 

-  Analysis  of  Alternatives 

-  Joint  Mission  Thread  (if  available) 

-  Joint/Service  Doctrine/CONOPS 
SME  input 
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Task  Description 


/ - \ 

( Decompose  the  mission  into  rehevant  tashs. 

\ _ / 

/  \ 

•  Mission(s)  decomposed  to  tasks  (activities)  and  sub-tasks  with 

key  nodes  identified 

-  Functional  performers/roles  identified  that  perform  the  mission 

-  Nodes  identified  as  the  “means”  to  performing  tasks  (e.g.  from 
DoDAF  products,  joint  mission  threads,  joint  publications, 
CONOPS) 

-  UJTLs  and  Service  task  lists  may  be  included 

V _ _ _ J 


IVJJSSJDJJ 

L»eserlprjDJj 


Task 
Description 


IVJjSSJDJJ 

J|  T'jjsk 

AiTrlbitlss 


iVimmfi 


fiiasis. v ^ 


lyMriaihraa 


rl'  rwtixbWYvj: 
Sysii;uu->T’i]sk 

->  iVijSSJDIi 
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Task  Description 

Example 

/ - \ 

( Decompose  the  mission  into  reCevant  tashs. 
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Mission  &  Task  Attributes 


Identify  the  attributes  (characteristics)  of  desired  effects  and  tashy. 


(  " 

•  Important  and  relevant  characteristics  of  Desired  Effects  and 

Tasks  are  identified 

•  JCIDS  prioritized  list  of  capability  attributes  for  enabling  JCAs 
(the  SWarF  list:  Battlespace  Awareness,  C2,  Net-centric, 
Logistics) 

•  Dimensions  of  performance  attributes  (time,  space,  quality, 
action,  etc)  are  directly  linked  to  task  and  sub-task  descriptions 

V _ _ _ ) 


*  TrneenMLiry; 
N  Sysisi-u-y'l'ijsii 
-y  Mission 


MjSSJOn 
.Des  crip  Mon 


TiisJs^ 
Les  trip  Mon 


Mission 

,  a 


Tiisis> 


!  Vi  ensures 
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Mission  &  Task  Attributes 

Example 


Identify  the  attributes  ( characteristics j  of  desired  effects  and  tashs. 


Task/Sub-Task  Attributes 

Operational  Task/Sub-Tasks 

Accurate 

Timely 

Networked 

Lethality 

A5.  Engage  Mobile  Target 

Vsfsfsfsfsfsfsfsfs 

11111111 

A51.  Release  Weapon 

X 

X 

A52.  Navigate  to  Target 

X 

X 

A53.  Acquire  and  Track 

X 

X 

A54.  Impact  Target 

X 

X 

A6.  Assess  Effectiveness 

X 

X 

Matrix  #5:  Operational  Task/Sub-Tasks  vs  Task/Sub-Task  Attributes 

Table  showing  SDB  task/sub-tasks  vs  attributes 
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Mission  &  Task  Measures 


(Ensure  there  are  separate  measures  for  the  military  effect  (mission 
accomplishment),  tas  ^performance,  and  system  function. 


•  Mission  Measures 

-  Should  assess  an  attribute  of  a  desired  effect 

-  Consists  of  a  scale  and  a  description 

•  Task  Measures 

-  At  least  one  measure  for  each  task-attribute  pairing 
(more  may  be  required) 

-  In  addition  to  JCIDS,  may  come  from  the 
joint/service  task  lists  (UJTL,  UNTL,  etc) 


Ti'iieejiLiJiryi 
Sysriau-./J'iJi.k 
->  Mjssjdij 


iVJjSSJDIi 
&  ITaais 
Mfimnrea 


iVJjSSJDJJ 

UBSfij'jpiJDiJ 
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Mission  &  Task  Measures 

Example 

(Ensure  there  are  separate  measures  for  the  military  effect  (mission 
accomplishment),  tas  ^performance,  and  system  function. 


Objectives:  1)  Achieve  battlefield  effects  against  Mobile  Targets  2)  Minimize  Undesired  Effects 

Measures 

Attributes 

Percent  of  actions  where  CAS 
was  employed  and  effectively 
reduced  risk  to  attack 

Time  to  effectively 
conduct  CAS  against 
mobile  targets 

Percent  of  CAS  missions 
where  fratricide  (including 
bodily  harm)  occurred  as  a 
direct  result  of  CAS 

Percent  of  CAS  missions 
where  collateral  damage 
from  CAS  was  acceptable 

Percent  of  systems 
integration  that  are 
successful 

Precision 

X 

X 

X 

Lethality 

X 

Timeliness 

X 

Flexibility 

X 

X 

Matriv  HpiiirpH  Attrihntp<;  iiq  Mpa«:i irp<: 

Task/Sub-Task  Measures 

Attributes 

Impact 

distance  to 

center  of 

target 

Probability 
of  Single 
Shot  (Pssk) 

Pet  post 

release  comm 
acknowledge 

Pet 

successful 

target 
updates  &. 
retargeting 

Pet 

successful 

weapon 

location 

updates 

Pet 

successful 

weapon 

aborts 

Avg  delta  in  actual 
target  location  & 
weapon  data  target 
location 

Pet 

accurate 

Bomb  Hit 

Indication 

reports 

Avg  time  to 
correctly 

assess  BDA 

A52.  Accuracy 

X 

A53.  Accuracy 

X 

A 54.  Accuracy 

X 

X 

A61.  Accuracy 

X 

A61.  Timeliness 

A51.  Networked 

A52.  Networked 

A53.  Networked 
A54.  Lethality 

X 

X 

X 

X 

X 

X 

X 

X  x— 

Mjssjdjj 


Tusjs 
IDesfirlptioju 


Mission 
&  'i'mk 
Aiirjjux-a 


Tni  ^ability: 
Mission 
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Mission/T  ask/  System 
Traceability 

(Ensure  system  attributes  are  tracea6Ce  to  tasf^ancf  mission. 

v _ / 

/  \ 

•  The  ICD  has  a  Capability  Gap  table  that  connects  capabilities 
(task-based)  with  attributes  and  metrics  (standards  for 
assessment) 

•  Look  for  a  connection  between  the  measured  system  attributes 
(KPPs,  KSAs,  others)  and  the  capability  gap  as  expressed  in 
the  ICD 


iVJjSSJDiJ 


IVJJSJUDU 


'I'iJSls 


Traceability: 
System-»Task 
-»  Mission  I 


Mjaatti) 


y.  r 


D 


pijoti 
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Mission/T  ask/  System 
Traceability  Example 


(Ensure  system  attributes  are  tracea6[e  to  tas f^ancf  mission. 


Priority 

Tier  1  &  2 
JCA 

Description 

Measure 

Minimum 

Value 

KPP,  KSA, 
other 
attributes 

1 

e.g.  Force 
Application 
Engagement 

Capability  1 

—Attribute  1 

Description 

Value 

6.1,  6.2 

—Attribute  n 

Description 

Value 

6.2,  6.3,  n 

2 

e.g.  Force 
Application 
Engagement 

Capability  2 

—Attribute  1 

Description 

Value 

6.1,  6.2.  6.3 

—Attribute  n 

Description 

Value 

6.4,  6.5,  n 

ICD  Capability  Gap  table  with  system  attributes  traced  to  task. 


Tiisii 


Traceability: 
System-»Task 
-» Mission  I 
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Mission  Decomposition 

Benefits 

•  Disciplined,  repeatable,  and  sufficient  process  for  developing 
mission,  task,  and  system  measures  for  testing 

-  The  Measures  Development  SOP  provides  this  process 

•  Enables  objective  understanding  of  a  system’s  contribution  to 
the  SoS,  task  performance,  and  mission  effectiveness 

•  Provides  traceability  to  warfighter  requirements 

•  Enables  validation  of  capability  gap  closure 

•  Moves  the  focus  from  “system  only”  to  task  and  mission 

•  Helps  design  tests  in  accordance  with  the  mission 

•  An  enabler  for  Design  of  Experiments 

•  Enhances  mission-based  test  design 

Enables  sufficient  conclusions  on  combat  mission  effectiveness 
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•  Introduction  and  Terminology 

•  Data  Collection 

•  Data  Analysis 

-  Characterizing  Each  Fragment 

-  Characterizing  Events 

•  Conclusions 
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Hazards  to  Dismounted  Troops  from 
Active  Countermeasures 


Example  Vehicle  Active  Countermeasures  (ACMs). 


Reactive  Armor 


Active  Protection  System  (APS) 


Primary  injury  mechanisms  for  dismounted  personnel: 


-  Penetrating  fragments 

•  Injuries  range  from  superficial  skin 
penetration  to  maximum  levels  of  trauma 

•  Severity  depends  on  size,  density,  velocity, 
and  shape  of  penetrating  fragments 

-  Blast  overpressure  (BOP) 

•  Lung  damage 

•  Eardrum  rupture 

-  Blunt  trauma 


Blast  Overpressure 


Thermal 


-  Thermal  energy 
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Terminology 
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Casualty  (Joint  Pub  1-02): 


Any  person  who  is  lost  to  the  organization  by  reason  of  having  been  declared  dead,  duty  status  -  whereabouts 
unknown,  missing,  ill,  or  injured 


Injury: 

Defined  with  the  Abbreviated  Injury  Scale  (AIS)  ©  *.  AIS  is  an  anatomically-based,  consensus-derived, 
international  severity  scoring  system  that  classifies  each  injury  by  body  region  according  to  its  relative  severity 
on  a  6-point  ordinal  scale.  AIS  scores  each  single  injury.  For  multiple  injuries,  Maximum  Abbreviated  Injury 
Score  (MAIS)  is  used  as  a  anatomical  measure  of  injury  severity.  The  MAIS  is  between  0  and  6. 

Our  threshold  of  unacceptable  risk  for  crew  and  dismounted  troops  is  a  serious  injury  (AIS3).  A  serious 
injury  is  one  that  requires  immediate  medical  attention.  Untreated  serious  injuries  could  cause  deterioration 
resulting  in  loss  of  life. 

Our  threshold  of  unacceptable  risk  for  civilians  is  a  minor  (AIS1)  or  moderate  (AIS2)  injury. 

Minor/moderate  injuries  range  from  superficial  to  those  that  are  fully  reversible  given  medical  attention  and 
pose  little  threat  to  loss  of  life. 

Personnel  who  exceed  these  thresholds  of  unacceptable  risk  would  be  considered  a  medical  casualty. 

Incapacitation: 

The  inability  to  perform,  at  a  level  required  for  combat  effectiveness,  the  physical  or  mental  tasks  required  in  a 
particular  role  at  a  specific  time  after  wounding.  Incapacitated  personnel  are  impaired  to  a  level  below  minimal 
capabilities  and  are  considered  an  operational  casualty. 

*  Abbreviated  Injury  Scale  ©  2005  Updated  2008,  AAAM,  Des  Plaines,  IL,  2008. 
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Casualty  (Joint  Pub  1-02): 

Any  person  who  is  lost  to  the  organization  by  reason  of  having  been  declared  dead,  duty  status  -  whereabouts 
unknown,  missing,  ill,  or  injured 

Injury: 

Defined  with  the  Abbreviated  Injury  Scale  (AIS)  ©  *.  AIS  is  an  anatomically-based,  consensus-derived, 
international  severity  scoring  system  that  classifies  each  injury  by  body  region  according  to  its  relative  severity 
on  a  6-point  ordinal  scale.  AIS  scores  each  single  injury.  For  multiple  injuries,  Maximum  Abbreviated  Injury 
Score  (MAIS)  is  used  as  a  anatomical  measure  of  injury  severity.  The  MAIS  is  between  0  and  6. 


Threshold  of  Unacceptable 
Risk  for  Non-Combatants 


Threshold  of  Unacceptable 
Risk  for  Dismounted  Troops 


Personnel  who  exceed  these  thresholds  of  unacceptable  risk  would  be  considered  a  medical  casualty. 

Incapacitation: 

The  inability  to  perform,  at  a  level  required  for  combat  effectiveness,  the  physical  or  mental  tasks  required  in  a 
particular  role  at  a  specific  time  after  wounding.  Incapacitated  personnel  are  impaired  to  a  level  below  minimal 
capabilities  and  are  considered  an  operational  casualty. 

March  2011  *  Abbreviated  Injury  Scale  ©  2005  Updated  2008,  AAAM,  Des  Plaines,  IL,  2008.  £ 


MAIS 

Injury  Level 

Head  Injury  Example 

Type  of  Injury 

0 

None 

No  injury 

None 

1 

Minor 

Minor  laceration  of  scalp 

Superficial 

2 

Moderate 

Major  laceration  of  scalp 

Reversible  injuries;  medical  attention  required 

3 

Serious 

Fracture  of  skull 

Reversible  injuries;  hospitalization  required 

4 

Severe 

Depressed  skull  fracture,  penetration  >  2  cm 

Non-reversible  injuries;  not  fully  recoverable  without  medical  care 

5 

Critical 

Depressed  skull  fracture,  laceration  of  spinal  artery 

Non-reversible  injuries;  not  fully  recoverable  with  medical  care 

6  Maximal  Massive  brain  stem  crush  Virtually  Unsurvivable 
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Objectives  for 
Testing  and  Analysis 


•  Design  cost-effective  tests  to  capture  primary  damage  mechanisms 

•  Define  collateral  hazards  in  terms  of  Probability  of  Injury  (P(l))  -  given  a  shot, 
given  a  hit,  etc. 

•  Quantify  and  assess  the  hazards  from  penetrating  insults  and  BOP  for  1 )  the  ACM, 
2)  the  threat,  and  3)  the  interaction  of  both 

-  Map  fragment  spray  of  CM  and  threat 

-  Determine  probability  of  injury  as  a  function  of  distance  from  detonation  point 

-  Determine  where  injury  potential  becomes  negligible 

•  Compare  hazards  caused  by  different  ACM  solutions 

•  Caveats: 

-  Based  on  a  limited  number  of  test  events  (typically  time  and  funding  do  not  permit 
statistically  strong  test  matrices) 

-  Limited  to  the  test  conditions 

-  Not  a  safety  assessment 
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Test  Configuration 


Full-Scale 

Arena 

Events 


Modified 

Arena 

Events 


Threat 


Not  to  Scale 


ACM 

T 

Vehicle 

surrogate 

Top  view 


Fragment 

trajectories 
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Data  Analysis: 
Characterizing  BOP 


BOP  Instrumentation: 

•  BTD 

-  Arranged  inside  the  vehicle 

-  Arranged  outside  the  vehicle  if  blast  is  main  damage  mechanism 

•  Free  Field  Blast  Pressure  Probes 

-  Used  in  static  arena  tests 

-  Used  in  dynamic  flight  tests  if  there  is  a  high  likelihood  of 
fragmentation 

•  Arranged  outside  of  the  vehicle 


•  Assumed  linear  trajectories  from  single  point  of  origin 


Operational  Requirement-based  Casualty 
Assessment  (ORCA)  Model: 

•  Embedded  INJURY  8.2  model  used  to  predict  lung  tissue 
damage 

•  Embedded  Department  of  Energy  (DoE)  auditory  injury 
criterion  used  to  predict  ear  drum  rupture 

•  Pressure-time  history  traces  from  each  location  are  used  as 
inputs 

•  Personnel  are  modeled  without  hearing  protection  facing  the 
direction  of  blast 


BTD  # 
Pressure  probes  4) 
Threat launch/aim  points 

Threatflightpath - 

CMflightpath - 
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Data  Analysis: 

Characterizing  Fragment  Effects 


ORCA  Model: 


Inputs: 


•  Uses  fragment  characteristics  and  the  simulated  dismounted  troop  properties 


-  Fragment  properties: 

•  Mass 

•  Striking  velocity 

•  Shape  factor 

•  Material  density 


-  Personnel  properties: 

•  Posture  (i.e.,  standing,  kneeling,  prone) 

•  Unarmored  (without  Personal  Protective 
Equipment  (PPE))  and  Armored  (With  PPE) 

•  Job  or  Combat  Role 


Assumptions: 

•  Probability  of  hit  =  1 

•  Hit  could  have  occurred  anywhere  on  the 
body 

Outputs: 

•  Models  injuries  caused  by  each  shot  line 
to  provide  severity  characterization  from 
each  fragment 

-  Probability  of  a  serious  or  greater 
injury  given  a  hit  (P(MAIS>3 1  hit)) 

-  Probability  of  a  moderate  or  greater 
injury  given  a  hit  (P(MAIS>2 1  hit)) 

•  Models  incapacitation  for  a  particular 
combat  role  at  a  given  post-wounding  time 


Fragment  C 
P(MAIS>3 1  hit)  =  0.8 


r~MAIS 

Injury  Level 

L  0 

None 

■  1 

Minor 

1  2 

7 3 

Moderate 

Serious 

■  4  Severe 

1  5 

Critical  1 

Injury  plots 
modeled  using 
same  fragment 
threat  conditions 
with  a  uniform 
grid  of  shot  lines 
in  a  front-only 
view  with  zero 
degrees  azimuth 
and  elevation 


Notional  results 


FragmentA 
P(MAIS>3 1  hit)  =  0.2 


Fragment  B 
P(MAIS>3 1  hit)  =  0.5 
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Number  of  Serious  or  Height  of  Height  of 

Greater  Injuries  Fragment  Impact  Fragment  Impact 
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Example  Analysis: 

Characterizing  Dispersion  and  Areas  of  Concern 


Notional  results 


Fragment  Dispersion 


Degrees 

Areas  of  Most  Concern 


Degrees 
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P(MAIS>3 1  hit)  P(MAIS>3 1  hit) 


-  Troop  target  arrays: 

•  Polar  array  around  detonation  point 

•  Grid  array 

•  Concentric  array:  facing  detonation,  along  an  180  degree  arc 

•  Custom  array  variables: 

-  Soldier  postures 

-  Distances  away  from  ACM-threat  interaction 

-  PPE  protection  levels 

-  Level,  slanted,  or  uneven  terrain 

•  Models  each  event’s  discrete  trajectories 

•  Models  velocity  retardation  from  air  drag  for  each  distance 

•  Computes  an  injury  or  incapacitation  level  as  a  result  of  each 
fragment  trajectory  at  each  distance 


!  in™1 


Example  Grid  Array 
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Example  Analysis: 
Characterizing  Each  Trajectory 


Notional  results 


r~MAIS 

Injury  Level 

L  0 

None 

■  1 

Minor 

1  2 

3 

Moderate 

Serious 

4 

Severe 

5 

March  2011 


The  color  of  the  trajectory  line  correlates 
to  a  MAIS  value 

Figure  illustrates  distances  where  injury 
potential  becomes  negligible 

-  1  serious  or  greater  injury  occurred 
at  4x  distance  (circled  in  red) 

-  0  serious  or  greater  injuries 
occurred  at  5x  distance 
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Example  Analysis  Continued: 
Characterizing  Each  Trajectory 


PSL 
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\  Data  Analysis: 

Characterizing  Hazards  Using  MUVES-S2/ORCA 


Uses  fragment  characteristics,  fragment  trajectories,  and  a 
dismounted  troop  target  array  as  inputs 

-  Fragment  trajectories: 

•  Impact  locations  on  panels  from  discrete  events 

•  Z-data  file  from  arena  test  of  ACMs  and/or  threats 

-  Troop  target  arrays: 

•  Typical  arrays  are  grid  arrays 


•  Each  dismounted  Soldier  in  the  array  is  independently  evaluated  for  every 
cell 

•  Custom  array  variables: 

-  Soldier  postures 

-  Distances  away  from  ACM-threat  interaction 

-  PPE  protection  levels 

-  Level,  slanted,  or  uneven  terrain 

-  Job  or  Combat  Role 

•  Computes  an  injury  or  incapacitation  level  at  various  distances 

•  Sample  analysis  outputs: 

-  MAIS  value 

-  Probability  of  a  serious  or  greater  injury  given  a  hit 

(P(MAIS>  3 1  hit)) 

-  Probability  of  a  minor  or  greater  injury  given  a  hit 

(P(MAIS>  1 1  hit)) 

-  Probability  of  hit 

-  Vehicle  outputs:  Pk,  Fkill,  Kkill,  Mkill 

March  2011 
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Example  Target  Array 


Target:  Dismounted  Threat:  Fragmenting 

Personnel  munition 
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Example  Analysis: 
Characterizing  Events  Examples 


RL 


Notional  results 


Low  Resolution  -  Large  Grid  High  Resolution  -  Small  Grid 


P(MAIS>X) 

0% 

0-5% 

15-10% 
10-15% 
15-20% 
20-25% 
25-30% 
30-35% 
35-40% 

1  40-45% 
45-50% 
50-55% 
55-60% 
60-65% 
65-70% 

75-80% 

80-85% 

85-90% 

90-95% 

95-99% 

99-100% 
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Example  Analysis: 
Summary  of  Collateral  Hazards 


RL 


Notional  results 


Areas  of  Concern  Given  an  Ideal  Intercept 


Worst  Case  Ranges 


Range  (m) 


MAIS 

Hazard  Area 

<m2) 

Max  Distance 
(m) 

1 

X 

a 

3 

Y 

p 

Max  CM-Threat 
Intercept  Range  (y) 


O  bop 

%  CM-Threat  Intercept  Point 


>  50%  Chance  of  Minor  Injuries 

>  50%  Chance  of  Serious  Injuries 

(MAIS  1) 

(MAIS  3) 
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Conclusions 


•  Active  counter  measures  present  hazards  to  dismounted  troops  and  non- 
combatants  in  the  vicinity  of  ACM-equipped  platforms 

•  The  U.S.  Army  Research  Laboratory  has  developed  methodology  using  ORCA 
and  MUVES-S2  to  characterize  these  collateral  hazards 

•  Collateral  hazard  results  may  be  used  to: 

-  Compare  hazardous  areas  between  ACM  solutions  to  assist  with  acquisition 
decisions 

-  Develop  Tactics,  Techniques,  and  Procedures  (TTPs)  for  combined  arms 
operations  requiring  dismounted  Soldiers  to  work  near  ACM-equipped 
platforms 

-  Assist  commanders  deploying  ACM-equipped  platforms  in  MOUT  operations 
near  civilian  populations 
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APG,  MD 

patricia.frounfelker@  us.army.mil 
410-278-3234  (DSN  298) 

Gregory  Dietrich 
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•  ORCAis  a  high-resolution  computerized  human- 
vulnerability  model  that  is  used  to  assess  the  impact 
of  various  casualty-causing  insults  on  personnel. 

•  ORCA  calculates  several  injury-severity-trauma 
metrics  that  may  be  used  to  characterize  both  an 
individual  injury  as  well  as  multiple  injuries  to  a 
single  person. 

•  ORCA  is  used  to  assess  the  impact  of  various 
casualty-causing  mechanisms  on  the  ability  of 
military  personnel  to  perform  battlefield  tasks. 

•  It  considers  the  operational  tasks  that  personnel  must 
perform,  and  determines  the  extent  to  which 
penetration  and  other  battlefield  insults  degrade  the 
ability  to  perform  these  tasks. 

•  The  model  can  be  applied  to  personnel  occupying  any 
crew  position  and  posture  on  any  combat  platform. 

•  Based  on  a  given  insult  or  set  of  insults,  ORCA 
assesses  whether  personnel  become  impaired  to  the 
extent  that  the  person  is  incapacitated  based  on  his 
specific  job/military  occupational  specialty  (MOS). 


Blast  Overpressure 


Acceleration 


Gall 
Bladder 


Permanent _  _ 

Cavity 


fci- —  Stomach 

# _ Subcutaneous 

Tissue 
* —  Skin 

Small  Intestine 
Liver 


Penetrating  Fragment 
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MUVES-S2  and  ORCA 


A  Survivability/Lethality/Vulnerability  (SLV)  computer  model  capable  of  analyzing 
the  effects  of  one  or  more  munitions  against  aircraft  or  ground-mobile  targets. 


m. 


ORCA  Methodology 

allows  for: 

•  discrete  shot  lines  through 
anatomy  based  on 
orientation  of  threat  trajectory 
to  personnel 

•  projectile  penetration 
mechanics  through  various 
a  n  atom  i  c  stru  ctu  res 

•  velocity  retardation  of  threat 
through  wound  track 

•  injury  description  by  type, 
severity,  and  frequency 

•  in-depth  description  of 
operational  effectiveness 


Threat 

Characterization 


-  r  ■  ~ 


Behind-Armor 

Debris 


Target 

Geometry 


Crew 

Casualty 


Vehicular  SLV 
Analysis 


i 


Component 
Defeat  Criteria 


Criticality  Analysis 
of  Components 
and  Subsystems 


Analysis  Outputs 

■  personnel  injury  and 
incapacitation 

■  system-level  kills /loss 
of  function 

■  residual  penetration  & 
velocity 

■  component  damage 

■  subsystem  capabilities 

■  remaining  system  utility 

■  user-defined  criteria 

■  tabu  lar&  graphical 
products 


Engagement 

Conditions 
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US  ARMY 


0  HDECOM 


Example  Analysis  Continued: 
Visualization  Examples 


Notional  results 


Hazard  Areas 


.2  50%  Chance 

2  50%  Chance 

250%  Chance 

2  50%  Chance 

of  Moderate 

of  Serious 

of  Severe 

ofCritical 

Injuries 

Injuries 

Injuries 

Injuries 

(MAIS=  2) 

(MAIS-3) 

(MAIS4) 

(MAIS5) 

March  2011 
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DT&E  Using  Scientific  T&E  Design 

George  Axiotis 

OSD,  Developmental  Test  &  Evaluation 
Deputy  Director,  Air  Warfare 

geroge.  axiotis@osd.  mil 

Mick  (Jedi)  Quintrall 

DT&E  AO  for  Airborne  Sensor  and  C2  Programs 

mickey,  quintrall.  ctr@osd.  mil 

NDIA  Conference 

March  15,  2011 


UNCLASSIFIED 


Scientific  T&E  Design 


Goal:  Understand  process  factors  and  their  interaction  with 
each  other,  so  that  their  results  produce  an  accurate  prediction 
of  the  outcome. 

-  Responses:  Desired/Expected  Outcomes 

-  Factors:  Important  Measures 


[FACTORS  are  — toad  categories  of  conditions  that  affect  responses,  ”  factors  + 
levels  =  operational  envelope,”  wrt  scientific  method:  responses  =  dependent 
variables,  factors  =  independent  variables,  the  MEASURES  should  be  chosen  before 
identifying  factors  and  levels] 


-  Levels:  Possible  Ranges/Extents  for  Factors 


Benefits  for  testers... 

-  Plan  based  on  statistical  confidence 


UNCLASSIFIED 


Deputy  Undersecretary  of  Defense  for 
Developmental  T&E’s  Position 


*  “Integrated  Testing  is  important  to  institute  in  order  to 
attain  test  data  that  can  be  used  across  the  acquisition 
processes...  Early  Planning  for  Integrated  Testing  sets 
up  complementary  individual  [DT  &  OT]  evaluation” 


*  STED  puts  discipline  into  T&E  planning. ..through 
structured  processes  such  as  “Design  of 
Experiments”.  STED  is  part  of  the  T&E  tool-bag  for 
OSD  Integrated  Testing  efforts” 


Integrated  Testing  and  Evaluation  Can  be  Aided  by  Applying  STED 
Methods  Across  Entire  Acquisition  Development  Cycle 


UNCLASSIFIED 


STED  in  DT&E  Planning 


*  Determine  Optimum  Test  Runs,  Test  Points  &  Resources 

-  Based  on  Factors,  Levels  &  Interactions 

-  Utilize  Statistical  Tools 

-  Forms  the  basis  of  Integrated  T&E 

*  Helps  Allocate  Test  Requirements  to  Test  Sequence 

-  Contractor  Test,  DT,  OT 

-  Component — Subsystem — System 

-  Informs  what  is  likely  to  be  learned  at  key  decision  points 

*  Iterative  Process 

-  Can  help  re-vector  test  plan  based  on  emerging  results 

-  Supports  better  use  of  Modeling  and  Simulation 


Reduce  test  time  and  statistically  consider  interactions  better  than  traditional 

one-factor-at-a-time  methods 


UNCLASSIFIED 


STED  in  the  Acquisition  Process 


a 


•  To  MSB 

-  Determine  what  functions  and  influences  are  most  important  to  T&E 
design  and  are  worth  close  monitoring 

-  Develop  the  T&E  test  space 

-  Identify  the  likely  T&E  resources  needs 

-  Supports  the  time-phasing  of  CT-DT-OT 

•  At  MS  C  [and  to  FRP] 

-  Assess  adequacy  of  T&E,  compared  to  data  accumulated 

-  Determine  future  T&E  priorities 

-  Identify  where  T&E  trades  can  be  made  given  results 


The  wise  investigator  expends  his  effort  not  in  one  grand  design  (necessarily 
conceived  at  a  time  when  he  knows  least  about  unfolding  reality),  but  in  a  series 

of  smaller  designs,  analyzing,  modifying,  and  getting  new  ideas  as  he  goes. 

—  G.  E.  P.  Box 


UNCLASSIFIED 


STED  Benefits  in  an  Integrated  T&E 

Environment 


•  Everyone  Understands  the  Test  Problem,  the  Test 
Environment  and  How  the  System  is  Tested 

•  Statistical  Tools  Identify  Optimum  Factors,  Test 
Points  and  Conditions  to  be  Tested 

•  Performance  being  Assessed  is  Allocated  to 
Specific  Tests  in  Sequence 

•  Allows  Comprehensive  Body  of  Data  to  be 
Accumulated  to  Support  Findings 

•  Facilitates  Coordination  of  Test  Events 


DT  Results  Better  Support  OT  Findings ,  Helping  Scope  OT 


UNCLASSIFIED 


STED  in  Use 


•  TEMP  -  DT&E  Expectations 

-  Part  III  -  Discuss  the  analytical  methodology  used  to  develop  the 
DT/IT  test  program 

-  Part  III  -  Show  the  Test  and  Evaluation  framework  in  chart  form 

-  Part  IV  -  Ensure  test  resources  are  mapped  to  the  T&E  framework 


•  Program  examples: 

-  SDB-II,  JAGM,  AIM-9X,  JASSM 

-  Examine  the  power  of  contractor  test  plans 

-  Develop  a  robust  (power/confidence)  integrated  test  approach 
CT/DT/OT  with  the  minimum  number  of  tests 

-  Recognize  scope  of  viable  testing  to  support  MS  C 

-  P-8,  AWACS,  J STARS,  F-35,  MQ-9 


UNCLASSIFIED 
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Back-up  Slides 


UNCLASSIFIED 


DoD  Policy  on  Integrated  T&E 


DoDD  5000.01: 

*  “Test  and  evaluation  shall  be  integrated  throughout 
the  defense  acquisition  process” 

DoDI  5000.02: 

*  “Integrate, . . .  successive  periods  of  DT&E,  LFT&E, 
and  IOT&E 

DoDI  5000.02,  Enclosure  2: 

*  “Developmental  and  operational  test  activities  shall  be 
integrated  and  seamless  throughout  the  phase” 

*  “Evaluations  shall  take  into  account  all  available  and 
relevant  data  and  information  from  contractor  and 
Government  sources” 


UNCLASSIFIED 


A 


DOE  Resources  for  Testers 


•  USAF  DoE  Community  of  Practice 

-  Web-ex  Mondays  1400  CT 

-  Contact:  https://connect.dco.dod.mil/eglindoe 
Gregory  T.  Hutto:  Gregory.Hutto@Eglin.af.mil 

•  Design  and  Analysis  Of  Experiments,  6th  Ed.,  2004 

-  Douglas  C.  Montgomery,  ISBN  0-471-15746-5 

•  Design  of  Experiments,  2nd  Ed.,  1957 

-  Cochran  and  Cox,  Wiley  and  Sons 

•  Response  Surface  Methodology ,  Process  and  Product  Optimization  Using 
Designed  Experiments,  3rd  Ed.,  2009 

-  Raymond  H.  Myers  and  Douglas  C.  Montgomery 

•  Joint  Test  and  Evaluation  Program  Handbook 

-  DOT&E,  December  2008 

•  Efficient  Simulation  Using  DOE  Methods 

-  Dr.  Tom  Donnelly,  SAS  Institute:  Tom.Donnelly@jmp.com 

•  Sample  Size,  Confidence  and  Designed  Experiments 

-  Dr.  Mark  Kiemele,  President,  Air  Academy  Associates:  aaa@airacad.com 


UNCLASSIFIED 


DOE  in  U.S.  DoD  T&E — Army 


a 


•  ATEC 

-  DoE  used  for  planning  system  evaluations  and  individual 
data-collection  events 

-  Single  table  depicts  how  the  individual  test  events  will 
manage  each  factor 

-  Be  able  to  reconfigure  for  unforeseen  events 

-  Manage  tradeoffs  between  operational  realism  and 
sufficient  data 

-  Requires  detailed  front-end  planning 


UNCLASSIFIED 


DOE  in  U.S.  DoD  T&E — Navy 


a 


•  COMOPTEVFOR 

-  DOE  part  of  Mission-based  Test  Design  (MBTD) 

-  A  shift  functional-based  to  mission-based  OT. 

-  OT  team  provides  detailed  OT  input  earlier  in 
program  schedule. 

-  OT  designed  around  factorial  design 

-  Sharing  of  T&E  responsibility,  resources,  and  data 
throughout  system  development. 

-  IOT&E  as  mission  capability  confirmation. 


UNCLASSIFIED 


DOE  in  U.S.  DoD  T&E —Air  Force 


•  53rd  Test  Wing 

-  With  digital  simulations,  screen  15-20  variables  with 
fractional  factorials  and  predict  performance 

-  In  HWIL,  confirm  digital  prediction  (validate  model) 
and  further  screen  8-12  factors;  predict 

-  In  live  fly,  confirm  prediction  (validate)  &  test  3-5  most 
vital  variables 

-  Prediction  Discrepancies  provide  opportunity  to 
improve  simulations 


UNCLASSIFIED 


Common  Range  Integrated  Instrumentation  System  (CRIIS) 


CRIIS  High  Accuracy  TSPI  Architecture 

and 


Technical  Maturity  Demonstration  Test 


Results 
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This  briefing  is:  UNCLASSIFIED 
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Outline 


■  CRMS  TSPI  Architecture  and  Algorithms 

-  RTK,  UTC,  SBAS,  and  RTK/UTC  Blending 

■  TSPI  Accuracy  Validation  Approach  and  Truth  Source 

-  M&S,  HIL,  Low  Dynamic  (Van,  Roller-Coaster),  High  Dynamics 

■  HIL  Simulation  Results 

■  Roller  Coaster  Test  Results  -  Lessons  Learned 

■  Flight  Test  Results  -  TRL6  Discussion 

■  Conclusion 


Rockwelf 
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Overview 


■  TSPI  Architecture  and  Algorithms 

■  TSPI  Accuracy  Validation  Approach 


Rockwelf 
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CRIIS  TSPI  Architecture  and 

Algorithms 


SBAS=  Precise  Corrections  for  SV  Position  and  Clock  Errors  (Sources:  StarFire,  JPL  etc) 


Rockwelf 
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TSPI  Level  II 

(/TCGPS^neit/aM(goi7fAmi 

■  Ultralight  Coupling  (UTC)  is  an  Essential  Part  of  High  Accuracy 
Positioning  in  a  High  Dynamic  Environment 

-  Reduces  TSPI  Error  Growth  by  Minimizing  Duration  of  GPS  Signal  Loss 

-  Signal  Re-established  Up  to  30  Seconds  After  Signal  Loss  without  the 
Need  to  Search 

■  Accurate  Relative  Timing  between  GPS,  Kalman  Filter,  and  IMU  is 
Essential  for  Highest  Accuracy  TSPI  Solution 

-  TSPI  Incorporates  Synchronous  Timing  between  GPS,  Processor,  and  IMU 

-  IMU  Strobe  is  Required  to  Minimize  Latency  Error  in  IMU  Measurements 
Used  to  Close  GPS  Signal  Tracking  Loops 

-  Minimizes  Error  Growth  Across  GPS  Outages 


TSPI  Level  II  GPS-Inertial  Design  Built  on  Core  UTC 
Approach  Successfully  Used  in  Phase  I  Demonstration 


Rockwefi 
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CRIIS  TSPI  Level  II  TSPI 
Verification  and  Truth  Sources 


Increasing  Fidelity 


Continued  Use  of  Crawl,  Walk,  Run  Approach  Used  in  Phase  I 
is  Proven  and  will  Continue  as  the  Verification  Model 


96ABW-201 1-0127 


Rockwell 
Collins 


Honeywell 


CRIIS  TSPI  Demo  Approach 

Incrementally  Phased 


Table  5  -  TSPI  Level  II  accuracy  requirements 


Position 

Horizontal 
(m  RMS) 

Position 
Vertical 
(m  RMS) 

Velocity 
Horiz/Vert 
(m/s  RMS) 

Acceleration 
Horiz/Vert 
(m/s2  RMS) 

Attitude 
(deg  RMS) 

Attitude  Rate 
(deg/s  RMS) 

Level  II  Real  Time 

0.3 

0.03 

0.03 

0.1 

0.2 

Level  II  Post  Processed 

0.1 

0.01 

0.01 

0.05 

0.1 

Test  Phase  Test  Objectives  Truth  Source 


Crawl: 

•  Model-Based 

•  Hardware-In-Loop 

•  Stationary 

Walk: 

Ground-Based 

Demonstrations 

•  Van 

•  Roller-Coaster 

Run: 

Flight  Demos 

•  T-38  Aircraft 


Validate  TSPI  solution 
accuracy  under 

•  GPS  simulation 

•  Live  Sky 

•  RRs  at  Various  Ranges 

Validate  TSPI  solution 
accuracy  under  low  and 
Moderate  dynamics 

•  RRs  at  Various  Ranges 


•  GPS  Simulator 

•  Surveyed  Antenna 


•  SPAN  (for  Position) 

•  Honeywell  EGI 
(For  Non-Positional 

TSPI  Parameters ) 


Validate  TSPI  solution 
accuracy  under  high  (flight) 
dynamics 

•  RRs  at  Various  Ranges 


•SPAN 

•  Honeywell  EGI 
(For  Non-Positional 
TSPI  Parameters ) 


Flgur*  10  -  AT  JJB  Talon 


Rockwell 
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HIL  Simulation  Results 


Rockwelf 

96ABW-201 1-0127  CoUtttS 
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Hardware-ln-The-Loop  Tests 


•  NavStorm+  GPS  Rx,  TSPI  Processor,  RT  Algorithms,  and  Simulated  HG-1700  IMU 

•  Spirent  Simulator  for  GPS  RF 

•  Antenna  Patterns,  Error  Models,  and  Simulated  Datalink  and  GPS  Outages 

•  Benefits: 

-  Perfect  Truth,  Identifies  Any  Algorithmic  Related  Common  Biases 

-  Lends  Credibility  to  Using  SUT-o-SUT  Comparison  when  10X  Truth  Not  Available 

Rockwell 

96ABW-201 1-0127  CoH/tlS 
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HIL  Simulation  of  50  nmi  Flight 

Trajectory 


•  Used  Actual  TSPI  Hardware  and  Software 

•  Atmosphere  and  IMU  Modeled  with  AMPSAT 

•  Used  T38  Antenna  Gain  Pattern 

•  Insensitive  to  Short  Datalink  Outages,  Loss  of  All  Reference  Receiver  Data,  and  SBAS 
Correction  Data  Outages 

•  Robust  to  Antenna  Phase  Effects 


RockweH 

96ABW-201 1-0127  CoUtHS 


HoneyweH 
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HIL  -  Nominal  50  nmi  Jet  Flight 


Segment# 

1 

2 

3 

4 

5 

6 

7 

Rqmt 

Units 

Acceleration  H 

0.031596 

0.029654 

0.030991 

0.028738 

0.031464 

0.031421 

0.035605 

0.03 

m/s/s 

Acceleration  V 

0.017828 

0.021379 

0.018959 

0.021449 

0.018222 

0.018126 

0.023477 

0.03 

m/s/s 

Velocity  H 

0.0062768 

0.015176 

0.01255 

0.0079937 

0.0057948 

0.0061308 

0.010889 

0.03 

m/s 

Velocity  V 

0.0043504 

0.0123 

0.0065374 

0.0054584 

0.0050342 

0.004428 

0.0062275 

0.03 

m/s 

Position  H 

0.044954 

0.12879 

0.10721 

0.12939 

0.24915 

0.18462 

0.055463 

0.3 

m 

Position  V 

0.25932 

0.25709 

0.2305 

0.18711 

0.08161 

0.13873 

0.11449 

0.3 

m 

Roll 

0.0086153 

0.0050646 

0.011735 

0.0057993 

0.0061491 

0.011675 

0.013846 

0.1 

deg 

Pitch 

0.0072549 

0.0071238 

0.0066403 

0.011235 

0.0072613 

0.019243 

0.013018 

0.1 

deg 

Heading 

0.015395 

0.010473 

0.016771 

0.017157 

0.027976 

0.027616 

0.019956 

0.1 

deg 

Roll  Rate 

0.01918 

0.019233 

0.018849 

0.018982 

0.019558 

0.019212 

0.024156 

0.2 

deg/s 

Pitch  Rate 

0.019092 

0.019313 

0.019156 

0.018596 

0.018828 

0.019171 

0.019229 

0.2 

deg/s 

Yaw  Rate 

0.018943 

0.019033 

0.01937 

0.019036 

0.019119 

0.018952 

0.019196 

0.2 

deg/s 

•  HIL  Test  Predicts  Good  TSPI  Performance  Even  with  Maneuvers  and  Long  Baseline 

•  Acceleration  Errors  Were  Large  Due  to  Lever  Arm  Amplification  and  IMU  Inertial  Sensor 
Assembly  Relative  Motion  with  Respect  to  Chassis 

-  Resolved  with  Use  of  Filtered  IMU  Outputs  for  TSPI  Acceleration 


Rockwelf 
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Roller  Coaster  Live  Test  Results 


Rockwelf 
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TSPI  Plate  Used  for  Demonstration 

Roller  Coaster  and  Flight  Tests 


PC-104/Data 


EGI 


Recorder  Stack 


SPAN 

IMU 


TSPI  GPS 
& 

Processor 


•  Two  CRMS  TSPI  Prototype  Systems  Used  for  Comparison  and  Consistency  Checking 

-  CRMS  TSPI  System  Under  Test  (SUT):  RCI  NavStorm+  GPS  Rx,  HG-1700  IMU  and  TSPI  Processor 

•  NovAtel  SPAN  Integrated  with  HG1700  for  Post-Mission  Reconstruction  of  Position  Truth 

•  Honeywell  HG-9900  Based  Embedded  GPS/INS  (EGI)  for  Non-positional  Truth 

•  GPS  Antenna  on  T-38  Aircraft  for  GPS  RF,  HAFB  L-Band  Antenna  for  Reference  Receiver  Datalink 

Rockwell  Hnnpvu/pll 
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Roller-Coaster  Maneuver  Segment 
for  Position  Accuracy  Analysis 


•  Blended  Position  Truth:  Fixed  Integer  SPAN  Position  Solution  and  Integrated  EGI  Velocity 

•  Position  Scoring  Segment  is  from  ‘Start  (top  of  first  hill)’  to  ‘Stop  (Plateau  of  Next  Hill)  Only 

-  Time  Duration  =  25  sec 

-  SPAN  Solution  Corrupted  for  Remainder  Segment  (Poor  GPS  Signals,  Multipath  etc.) 

•  Non-Positional  TSPI  Parameters  Scored  Over  Entire  Roller-Coaster  Trajectory 


Rockwelf 
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Roller-Coaster:  Blended  Truth 

Reference 


R1  TAR2 _ Dayl  RunB:  SRAM  Position  Uncertainty  (1  ) 


R1  TAP2JDay1RunB:  TSPI  -  Truth  Delta  Position 


<i>  0.15  - 


2100/  21SQ  2200  2210  2230  Z230 

Time  in  Seconds  from  GPS  time  277581 ,2623- 

Figure  1:  TSPI -SPAN 


SPAN  Truth  Corrupted:  Inconsistency 
Between  SPAN  Indicated  Position 
Uncertainty  and  SPAN  Position  Solution 
During  Non-RTK  Mode 
Blended  Position  Truth:  Used  Initial  Fixed 
Integer  SPAN  Solution  Propagated  by 
Integrated  EGI  Velocity 


-  East 

-  North 

-  Vertical 


21  SO  2100  2200  2210  2220  2230 

Time  in  Seconds  from  GRS  time  277581  .2623 


R1TAP2_Day1RunB_ModPosTruth:  TSPI  -  Truth  Delta  Position 


2180  2190  2200  2210  2220  2230 

Time  in  Seconds  from  GPS  time  277581 .2623 
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Common  Range  Integrated  Instrumentation  System  (CRIIS) 


Rockwell 
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Common  Range  Integrated  Instrumentation  System  (CRIIS) 


Rockwell 

96ABW-201 1-0127  COlf/ItS 


HoneyweH 


Common  Range  Integrated  Instrumentation  System  (CRIIS) 


Rockwell 
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Common  Range  Integrated  Instrumentation  System  (CRIIS) 


Rockwell 
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Common  Range  Integrated  Instrumentation  System  (CRIIS) 
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Dynamic  Flight  Test  Results 


Rockwelf 
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Reference  Receivers  and 
^^^)atalin]<Set^Ui^^ 


•  Four  Reference  Receivers  Spaced  -  20  nmi  Apart  at  Surveyed  Locations  (CORS  Used) 

•  Datalinks  Set  Up  at  Each  End  of  the  Range 

-  Used  for  Uplinking  SBAS  Corrections  and  DGPS  Measurements  from  RR 

•  Data  from  All  Four  RRs  Used  for  Producing  Truth,  Post-Mission,  Using  SPAN 

•  To  Accommodate  Short  and  Long  Baseline  Requirements  Data  from  One  Appropriate 
RR  Used  in  CRIIS  TSPI  Computation 


Rockwelf 
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CRIIS  TSPI  High  Dynamics 

Flight  Test  29  Oct  2009  -  Holloman  AFB 


Perform 


Flight  Profile  #1  Maneuvers 

Flight  Profile  #1  Maneuvers 

1  -  Split  S  To  Cuban  8 

9  -  Right  Aileron  Roll 

2  -  Orbit 

10  -  Straight  And  Level 

3  -  Climb 

11  -  Left  Aileron  Roll 

4  -  Straight  And  Level 

12  -  Straight  And  Level 

5  -  3G  Turn 

13-  Max  Accel 

6  -  Straight  And  Level 

14  -  Break  Turns 

7  -  Max  G  Turn 

15  -  Straight  And  Level 

8  -  Straight  And  Level 

16  -  Orbit 

☆ 


5-5 CS  &  Climb 
After  Q5 
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Delta  velocity  in  m/s 


TSPI A  to  B  Velocity  Differences 


TSPIB_FT_20091 029B:  TSPI-EGI  Delta  Velocity  NED  LAC 


0.2 

0.15  - 

0.1  - 

0.05  - 

0  - 

-0.05  - 

-0.1  - 

-0.15  - 


Velocity  Difference  TSPI  A  -  TSPI  B 


-0.2 


RMS  in  m/s: 

North  =0.0205 
East  =0.0221 
Down  =  0.0169 


IbmuLdnihi  i  Jill  I . 


600  600  1000  1200  1400  1600  1800  2000  2200  2400  2600 

Time  in  Seconds  from  GPS  time  419595.5731 


-0.2 


600  800  1000  1200  1400  1600  1800  2000  2200  2400  2600 

Time  in  seconds 


•  Horizontal  Differences  Between  TSPI  A  &  B  are  Less  Than  Half  Those  of  TSPI  B  and  EGI 

•  Lever  Arm  Errors  in  EGI  IMU-to-GPS  Antenna  Are  Suspected  Cause 
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Velocity  Accuracy 

Using  SUT-1  to  SUT-2  Difference 


T SPIA_FT_20091 02,9  b :  TSPI-EGI  Delta  Velocity  NED  LAC 


Time  in  Seconds  from  GPS  time  419595.5736 


ueiia  veiocny  oitu 


Key  Observations: 

•  Anomalies  in  CRMS  to  EGI  and  CRS  Velocity 

•  Differences  During  High  Rotation  Rate  Maneuvers 

-  Not  Common  Mode  CRMS  Errors,  Since 
Signatures  for  Each  Truth  Source  is  Different 

•  Anomalies  in  EGI  to  CRS  Velocity  Differences,  Much 
Larger  than  SUTs  Differences 

-  Source  of  Anomalies  is  Lever  Arm  Errors 

•  CRMS  TSPI-A  to  TSPI-B  Consistent 

-  Method  Can  be  Used  for  Accuracy  Verification, 
Along  With  HIL  (or  Other  Simulation  to  Verify  Lack 
of  Large  Common-Mode  Deterministic  Errors) 


TSPlA_FT_2DDfi 1029b:  Delta  Vel  TS PIA-TGP IE-  MED 
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Acceleration  Noise  Issue 


TSPIA  FT  20091029b:  TSPI-EGI  Delta  Acceleration 


TSPIB  FT  20091029b:  TSPI-EGI  Delta  Acceleration 


•  Vibrations  During  High  G  Maneuvers  Added  High  Frequency  Noise  to  Relative 
Acceleration  Between  EGI  and  TSPI 

•  Data  Must  Be  Filtered  to  Below  the  Shock  Roll-Off  Frequency  of  Each  Systems 

-  High  Frequency  Noise  and  Shock  mount  Resonance  Should  be  Above  Filter  BW 
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Common  Range  Integrated  Instrumentation  System  (CRIIS) 
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Delta  attitude  in  degrees 


Attitude  (Roll,  Pitch)  Accuracy 


TSPIB  FT  20091029b:  TSPI-EG I  Delta  Attitude  Local  Leve  TSPIB  FT  20091029b:  RMS  Delta  Attitude 


000  300  100D  1200  1400  1600  1  BOD  2DOO  2200  2400  2900  1000  1200  1400  1600  1800  2000  2200  2400 

Time  in  Seconds  from  GPS  time  419595.5731  Time  in  Seconds  from  GPS  time  419595.5731 


•  Attitude  Accuracy  Met  Requirements  with  Margin  Even  Under  High  Dynamics 

-  EGI  Used  as  Truth 

-  No  Filtering  Applied  for  Processing 

•  Segment  by  Segment  RMS  Values  are  Shown 
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Attitude  (Heading)  Accuracy 


T S P IA_FT_2Q09 1029b  TSPI-EGI  Delta  Heading  TSPIA_FT_20091 029b:  RMS  Delta  Heading 


•  Heading  was  Well  Aligned  After  Takeoff  Roll 

•  Heading  Accuracy  Maintained  During  Maneuvers  and  Straight  &  Level  Segments 

•  EGI  Used  as  Truth  (No  Filtering  Applied) 

-  RMS  Segment  Errors  Well  within  Spec 
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Delta  attitude  rate  in  deg/s 


Attitude  Rates  Performance 


TSPIB  FT  20091 023B:  TSPI  -  EGI  Delta  Rotation  Rates 


TSPIB  FT  20091029B:  RMS  Delta  Rotation  Rates 
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q: 


0.2 
0.18 
0.16 
0.14 
0.12 
0.1 
0.08 
0.06 
0.04 
0.02 
0 


100%:  Pass 


Time  in  Seconds  from  GPS  time  419595.5731 


1000  1200  1400  1600  1800  2000  2200  2400 

Time  in  Seconds  from  GPS  time  419595.5731 


•  Attitude  Rate  Performance  Good 

-  Data  Processed  with  1  Hz  Butterworth  to  Filter  Out  High  Frequency 
Relative  Motion  between  EGI  and  TSPI 


Rockwelt 

96ABW-201 1-0127  CoUtttS 


Honeywell 


30 


Flight  Test  Results 

TSPI  Level  II  Absolute  Mode  Summary 


•  Absolute  Mode  Positioning  Achieves  Significant  Margin 


Flight 

27  Oct, 

Flight  1 

27  Oct, 

Flight  1 

27  Oct, 

Flight  1 

27  Oct, 

Flight  1 

27  Oct, 

Flight  1 

Accuracy 

Rqmt 

Maneuver  Type 

Cuban  8 

180  2g  Turn 

360  3g  Turn 

360  5g  Turn 

360  Degree 
Aileron  Roll 

Maneuver  Segment 
(sec) 

1926-2045 

2292-2340 

2350-2415 

2540-2656 

2725-2743 

Horizontal  Position 
Accuracy  (m) 

0.3 

0.7 

0.6 

0.7 

0.7 

3.0  m 

Vertical  Position 
Accuracy  (m) 

0.9 

0.9 

0.7 

0.9 

1.9 

4.6  m 

Horizontal  Velocity 
Accuracy  (m/s) 

(TSPI  A-B) 

0.02 

0.01 

0.01 

0.04 

0.02 

0.05  m/s 

Vertical  Velocity 
Accuracy  (m/s) 

(TSPI  A-B) 

0.01 

0.01 

0.01 

0.01 

0.01 

0.05  m/s 

Baseline  (nmi) 

NA 

NA 

NA 

NA 

NA 

RTK  Mode 

4,2,0 

1 

1 

1,0 

1,0 

Note:  Velocity  Accuracy  Included  in  Absolute  Mode  as  It  is  a  Special  Case  Where  No  Datalink  is  Available 
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Flight  Test  Results 
TSP  Level  II  Position  Accuracy 


•  TSPI  Level  II  Position  Accuracy  Met  for  Low  and  High  Dynamic  Maneuvers 

-  Split-S  Not  Met  in  Post  Processing  but  is  Improved  Over  Real  Time 


Flight 

29  Oct, 
Flight  2 

29  Oct, 
Flight  2 

29  Oct, 
Flight  2 

29  Oct, 
Flight  2 

29  Oct, 
Flight  2 

29  Oct, 
Fit  3 

29  Oct, 

Fit  3 

29  Oct, 

Fit  3 

Rqmt 

(m) 

Maneuver  Type 

Climb 

360 

degree 

3g  turn 

360 

degree 

5g  turn 

360 

degree 

aileron 

roll 

Straight 
&  Level 

Split-S 

Straight 
&  Level 

50  degree 
roll;  180 
degree 
turn 

Maneuver  Segment 
(sec) 

1325- 

1422 

1770- 

1840 

1935- 

2072 

2143- 

2158 

1840- 

1935 

1990- 

2084 

3252- 

3393 

3393- 

3490 

Real  Time  Horizontal 
Position  Accuracy  (m) 

0.02 

0.1 

0.1 

0.1 

0.1 

0.2 

0.2 

0.1 

0.3 

Real  Time  Vertical 
Position  Accuracy  (m) 

0.1 

0.2 

0.1 

0.2 

0.1 

0.3 

0.1 

0.1 

0.3 

Post  Mission 

Horizontal  Position 
Accuracy  (m) 

0.02 

0.1 

0.05 

0.1 

0.1 

0.2 

0.1 

0.04 

0.1 

Post  Mission  Vertical 
Position  Accuracy  (m) 

0.02 

0.1 

0.1 

0.2 

0.05 

0.2 

0.1 

0.1 

0.1 

Max  Baseline  (nmi) 

1 

16 

10 

18 

6 

40 

50 

54 

50 

RTK  Mode 

8 

8, 5, 4, 2,0 

8, 7, 4, 2 

8,4,0 

8 

4 

4 

4,1 
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Flight  Test  Results 
TSPI  Level  II  Non-Position  Solution 


•  Attitude  Accuracy  Passes  with  Significant  Margin  in  Most  Cases 

-  TSPI  Level  II  Attitude  Accuracy  as  Scored  by  Holloman  CRS  in  this  Case 


Flight 

29  Oct, 
Flight  2 

29  Oct, 
Flight  2 

29  Oct, 
Flight  2 

29  Oct, 
Flight  2 

29  Oct, 
Flight  2 

Requirement 

(deg) 

Maneuver  Type 

Climb 

360  degree 
3g  turn 

360  degree 
5g  turn 

360  degree 
aileron  roll 

Straight  & 
Level 

Maneuver  Segment  (sec) 

1325-1422 

1770-1840 

1935-2072 

2143-2158 

1840-1935 

Real  Time  Roll  Accuracy 

(deg) 

0.03 

0.02 

0.02 

0.03 

0.1 

0.1 

Real  Time  Pitch 

Accuracy  (deg) 

0.01 

0.01 

0.01 

0.01 

0.1 

0.1 

Real  Time  Heading 
Accuracy  (deg) 

0.01 

0.01 

0.02 

0.02 

0.1 

0.1 

Post  Mission  Roll 
Accuracy  (deg) 

0.005 

0.01 

0.01 

0.01 

0.005 

0.05 

Post  Mission  Pitch 
Accuracy  (deg) 

0.01 

0.02 

0.01 

0.01 

0.005 

0.05 

Post  Mission  Heading 
Accuracy  (deg) 

0.01 

0.01 

0.01 

0.01 

0.01 

0.05 
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Flight  Test  Results 
TSPI  Level  II  Non-Position  Solution 


•  Attitude  Rate  Accuracy  Passes  with  Significant  Margin  in  Most  Cases 

-  TSPI  Level  II  Attitude  Accuracy  as  Scored  by  Holloman  CRS  in  this  Case 


Flight 

29  Oct, 

Flight  2 

29  Oct, 

Flight  2 

29  Oct, 

Flight  2 

29  Oct, 

Flight  2 

29  Oct, 

Flight  2 

Requirement 

(deg/sec) 

Maneuver  Type 

Climb 

360  degree 
3g  turn 

360  degree 

5g  turn 

360  degree 
aileron  roll 

Straight  & 
Level 

Maneuver  Segment  (sec) 

1325-1422 

1770-1840 

1935-2072 

2143-2158 

1840-1935 

Real  Time  Roll  Accuracy 
(deg) 

0.04 

0.04 

0.1 

0.1 

0.1 

0.2 

Real  Time  Pitch  Accuracy 
(deg/sec) 

0.03 

0.04 

0.2 

0.02 

0.1 

0.2 

Real  Time  Heading 

Accuracy  (deg/sec) 

0.1 

0.1 

0.1 

0.04 

0.2 

0.2 

Post  Mission  Roll 

Accuracy  (deg/sec) 

0.1 

0.1 

0.1 

0.1 

0.02 

0.1 

Post  Mission  Pitch 

Accuracy  (deg/sec) 

0.02 

0.03 

0.1 

0.02 

0.01 

0.1 

Post  Mission  Heading 
Accuracy  (deg/sec) 

0.05 

0.02 

0.1 

0.02 

0.01 

0.1 
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Flight  Test  Results 
TSPI  Level  II  Non-Position  Solution 


•  Uncorrelated  Mechanical  Vibration  Modes  between  TSPI  and  Truth 
Sources  Caused  Large  Velocity  Errors  at  Point  of  Navigation  (GPS 
Antenna  for  This  Test) 

-  Hence,  Truth  Sources  Were  Not  Capable  of  Scoring  the  TSPIs 


Flight 

29  Oct, 

Flight  2 

29  Oct, 

Flight  2 

29  Oct, 

Flight  2 

29  Oct, 

Flight  2 

29  Oct, 

Flight  2 

Requirement 

(m/s) 

Maneuver  Type 

Climb 

360  degree 
3g  turn 

360  degree 
5g  turn 

360  degree 
aileron  roll 

Straight  & 
Level 

Maneuver  Segment  (sec) 

1325-1422 

1770-1840 

1935-2072 

2143-2158 

1840-1935 

Real  Time  Horizontal 
Velocity  Accuracy  (m/s) 

0.01 

0.05 

0.07 

0.05 

0.01 

0.03 

Real  Time  Vertical  Velocity 
Accuracy  (m/s) 

0.01 

0.04 

0.07 

0.06 

0.04 

0.03 

Post  Mission  Horizontal 
Velocity  Accuracy  (m/s) 

0.015 

0.04 

0.06 

0.04 

0.01 

0.01 

Post  Mission  Vertical 
Velocity  Accuracy  (m/s) 

0.01 

0.02 

0.04 

0.04 

0.01 

0.01 
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Flight  Test  Results 
TSPI  Level  II  Non-Position  Solution 


•  TSPI  Level  II  Velocity  Consistency  between  TSPI  A  &  B  was  Investigated, 
Since  Truth  was  Severely  Impacted  by  Lever  Arm  Length 

•  Consistency  between  TSPI  Units  is  Very  Good  as  Seen  Below 

-  TSPI  A/B  Comparison  is  an  Indicator  that  Level  II  Velocity  Can  be  Met 


Flight 

29  Oct, 

Flight  2 

29  Oct, 

Flight  2 

29  Oct, 

Flight  2 

29  Oct, 

Flight  2 

29  Oct, 

Flight  2 

Requirement 

(m/s) 

Maneuver  Type 

Climb 

360  degree 
3g  turn 

360  degree 

5g  turn 

360  degree 
aileron  roll 

Straight  & 
Level 

Maneuver  Segment 
(sec) 

1325-1422 

1770-1840 

1935-2072 

2143-2158 

1840-1935 

Real  Time  Horizontal 
Velocity  Accuracy 
(m/s) 

0.005 

0.017 

0.04 

0.013 

0.004 

0.03 

Real  Time  Vertical 
Velocity  Accuracy 
(m/s) 

0.004 

0.009 

0.03 

0.008 

0.002 

0.03 

Post  Mission 

Horizontal  Velocity 
Accuracy  (m/s) 

0.003 

0.01 

0.01 

0.030 

0.002 

0.01 

Post  Mission  Vertical 
Velocity  Accuracy 
(m/s) 

0.003 

0.005 

0.01 

0.01 

0.001 

0.01 

Rockwell 
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Flight  Test  Results 
TSPI  Level  II  Non-Position  Solution 


•  High  Frequency  Motion  components  were  Aliased  to  Near  DC  in  50  Hz  TSPI  Acceleration  Outputs 

-  Primary  Driver  Was  Vibratory  Motion  of  Isolated  Inertial  Sensor  Assembly  Relative  to  IMU 
Chassis 

-  Problem  will  be  Addressed  in  CRMS  Phase-ll  via  Additional  Filtering  of  IMU  Outputs  Used  Only 
for  Generation  of  the  TSPI  Acceleration  outputs 


Flight 

29  Oct, 

Flight  2 

29  Oct, 

Flight  2 

29  Oct, 

Flight  2 

29  Oct, 

Flight  2 

29  Oct, 

Flight  2 

Requirement 

(m/s/s) 

Maneuver  Type 

Climb 

360  degree 
3g  turn 

360  degree 
5g  turn 

360  degree 
aileron  roll 

Straight  & 
Level 

Maneuver  Segment  (sec) 

1325-1422 

1770-1840 

1935-2072 

2143-2158 

1840-1935 

Real  Time  Horizontal 
Acceleration  Accuracy 
(m/s/s) 

0.04 

0.07 

0.18 

0.03 

0.02 

0.03 

Real  Time  Vertical 
Acceleration  Accuracy 
(m/s/s) 

0.03 

0.03 

0.08 

0.04 

0.02 

0.03 

Post  Mission  Horizontal 
Acceleration  Accuracy 
(m/s/s) 

0.01 

Post  Mission  Vertical 
Acceleration  Accuracy 
(m/s/s) 

0.01 

PMP  Acceleration  Could  Not  be  Scored  Due  to  Measurement  Aliasing 
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Flight  Test  Results 
TSPI  Level  II  Non-Position  Solution 


•  TSPI  Level  II  Acceleration  Accuracy  was  Evaluated  for  Consistency  between  TSPI  A  &  B 

-  Aliasing  Found  in  50  Hz  Data  and  Not  the  Recorded  300  Hz  Raw  IMU  Data 

-  300  Hz  IMU  Data  Used  to  Compare  TSPI  A  &  B 


Flight 

29  Oct, 

Flight  2 

29  Oct, 

Flight  2 

29  Oct, 

Flight  2 

29  Oct, 

Flight  2 

29  Oct, 

Flight  2 

Requirement 

(m/s/s) 

Maneuver  Type 

Climb 

360  degree 

3g  turn 

360  degree 

5g  turn 

360  degree 
aileron  roll 

Straight  & 
Level 

Maneuver  Segment  (sec) 

1325-1422 

1770-1840 

1935-2072 

2143-2158 

1840-1935 

Real  Time  Horizontal 
Acceleration  Accuracy 
(m/s/s) 

0.01 

0.01 

0.01 

0.01 

0.004 

0.03 

Real  Time  Vertical 
Acceleration  Accuracy 
(ml  sis) 

0.01 

0.02 

0.01 

0.01 

0.01 

0.03 

Post  Mission  Horizontal 
Acceleration  Accuracy 
(ml  sis) 

0.01 

Post  Mission  Vertical 
Acceleration  Accuracy 
(m/s/s) 

0.01 

300  Hz  IMU  Data  Could  Not  be  Used  in  PMP  as  it  Operates  Only  on  50  Hz  Data 
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Common  Range  Integrated  Instrumentation  System  (CRIIS) 
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Conclusions 


■  CRMS  High  Dynamic  Real-Time  TSPI  Developed  and 
Implemented  Using  State-Of-The-Art  Processing 
Algorithms 

■  CRMS  TSPI  Level-ll  Accuracies  Successfully 
Demonstrated  thru  a  Phased  Approach 

-  M&S,  HIL,  Van,  Roller-Coaster  Used  to  Identify  issues  and 
Tune/Fix  Algorithms 

-  High  Dynamics  Flight  Test  Results  Demonstrate  TRL6  Maturity 
(Performance  in  Relevant  Environment) 

■  System  Development  in  EMD  Phase 
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r 

J  Intent  must  be  ma 

L _ 

intained  throughout  the  proi 

grams  lifecycle  to  ensure  warfighter  need  is  provided 
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Pre-Milestone  A 
Pitfalls  &  Solutions 


NORTHROP  GRUMMAN 
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A  lack  of  clear  communication  between  those 
setting  requirements  and  those  in  the 
acquisition  process  turning  requirements  into 
acquisition  plans  and  contract  specifications. 

HOUSE  ARMED  SERVICES  COMMITTEE  PANEL  ON  DEFENSE  ACQUISITION  REFORM 
FINDINGS  AND  RECOMMENDATIONS  March  23,  2010 

“The  Act  fWSARA  2009]  recognizes  that 
‘unrealistic  performance  expectations’  and 
‘immature  technologies’  are  among  the  root 
causes  of  trouble  in  defense  programs,” 

DOT&E  Director  J.  Michael  Gilmore  Feb  24  2010  Aviationweek.com 


T.  early  stages  of  an  acquisition  program  are  In 

many  ways  the  most  critical.  It  is  in  the  early  stages 
that  investments  must  be  made  in  systems 
engineering,  in  acquiring  technical  data  rights  to 
support  competition  and  system  sustainment,  and 
in  robust  developmental  testing” 

HOUSE  ARMED  SERVICES  COMMITTEE  PANEL  ON  DEFENSE  ACQUISITION  REFORM 
FINDINGS  AND  RECOMMENDATIONS  March  23,  2010 


Solutions 


Fund  The  Program  Correctly 
Establish  A  robust  Systems  Engineering  Community 
Government  /  Industry  Working  Groups 
Technical  Reviews 
Coordinated  Capabilities-  ICD 
Contract  Language  Supporting  Integrated  Testing 


TES-Test  Strategy 


Starting  a  program  right  is  essential  to  program  success 
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Technology  Development  Phase 
Pitfalls  &  Solutions 
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Materiel 

Solution 

Analysis 

Technology 

Development 

Engineering  and 
Manufacturing 

Develop  ment 

Production  &  Operations  & 

Deployment  1  Support 

>  Development 
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-Systems  Acquisition^xS^ 


Sustainment  3^ 


.  Pre 


Systems  Acquisition 


<£>  =  Decision  Point  /\=  Milestone  Review  <  >  =  Decision  Point  if  PDR  is  not  conducted  before  Milestone  B 


‘....and  ‘immature  technologies’  are  among  the  root 
causes  of  trouble  in  defense  programs’ 

DOT&E  Director  J.  Michael  Gilmore  Feb  24  2010  Aviationweek.com 


Solutions 


Technology  Development  and  Risk  Assessment 
lacks  Connectivity  to  Operational  Mission 
“immature  technologies”  entering  EMD 

Former  DOT&E  Director  Charles  McQueary  Feb  2008  NDIA 


•A  sound  Technology  Development  Plan  (TRL 
Maturation) 

•A  coordination  between  the  TES  and  TEMP 
•A  capability  Development  Document  that  details  the 
operational  performance  parameters  for  the 
anticipated  system 

•Test  Planning  establish  Integrated  Test  Plans 
accounting  for  Risk  Related  Activities 
Implement  Integrated  Test  Concept  WIPT  /  CTF^yO^ 


Technology  Development  must  be  performed  within  the  intended  operational  environment 
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Engineering  and  Manufacturing 
Development-  Pitfalls  &  Solutions 
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Production  and  Deployment 
Pitfalls  &  Solutions 
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Materiel 
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Pre-Systems  Acq 


Prod  uct l«=»n  & 
Depl  oy  me  nt 


LRIP/IOT&E 


Systems  Acquisition 


FRP 

Decision 
Re  vi  ew 


'  I _ _ 

S  u  sta  inment 


Decision  Point  X\,=  Milestone  Review  ’>  =  Decision  Point  if  PDR  is  not  conducted  before  Milestone 

Solutions 


Requirements  Creep  -  preclude 
destabilizing  requirements  changes 
5000.02  Dec  08 

PROCESS  Charles  “Pete”  Adolph  NDIA  May  2010 


Incomplete  0 A  Assessments  Prior  to 
Authorization  to  LRIP  /  Production 

Former  DOT&E  Director  Charles  McQueary  Feb  2008  NDIA 


•Updated  TEMP  and  Comprehensive 
Capabilities  Product  Document  (CPD) 
•Contractor  Production  Plan  consistent  with 
CPD 

•ATP's  test  with  proper  environment  where 
applicable 

•Implement  Block  update  acquisition  policy 
"Evolutionary  Acquisition" 


0 


Build  what  you  Intended  -  No  More  -  No  Less 
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Lessons  Learned 


NORTHROP  GRUMMAN 


Program  Problem 

Issue 

Fix 

Incomplete  and 
Ambiguous 

Requirements 

Lack  of  Early  Program 

Skill  Mix  inside  SEIT 
prevented  complete 
requirements  Set 

Definition 

Introduce  and  Program  for  Complete  SEIT  Skill 
including  Specialty  Engineering  and  Test  and 
Evaluation  Personnel 

Establish  early  verification  program 

Risk  program  not 
aligned  with  realistic 
operational 
environments 

Technology  development 
inconsistent  with  needs 

Test  Plan  Integrated  with  SEP  established  Risk 
program  waterfalls,  planned  early 

Contractor  Test  Program  Integrated  with  TES  and 
TEMP 

Test  Plan  Intent  not 
used  during  EMD 

Test  Plan  not  maintained 
throughout  the  EMD  test 
program 

Collaborative  Test  Plan  Model  Maintained 
throughout  the  Program  Lifecycle  (SE  Model  Tools 
Used  to  support  Collaboration  and  Modeling) 
Contractor  Test  Plan  Aligned  with  Requirements 
Verification  and  TEMP  though  SE  traceability  tool  set 

Data  Rights  Not 
Negotiated 

Prevents  OA  using  early 
test  data 

Good  Contract  Language  and  Propriety  Information 
Agreements  Established  Early  in  a  Program 

Maintaining  the  Original  Intent  Delivers  the  Right  Product  to  the  Warfighter 
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Test  Verification  and  Planning  Guarding  the  Intent 
A  Contractor's  Look  at  A  Hybrid  Solution 


NORTHROP  GRUMMAN 


CDD,  Requirements,  TES 
TEMP,  Risk  Plan,  Product 
Architecture,  SEP,  Program 
Management  Plan 


I  ntegrated  Test  Team,  Verification 
Requirements,  Test  Unique 
Requirements,  Test  Planning,  Test 
Plan  Modeling  and  Schedules, 
Architectural  Refinement 


r 


o3 

£ 


c/5 

O 

o 

Ph 


C/5 

<D 


> 


iHl 


1  1 


•Test  Plan  Strategy  Developed  and  aligned  with  the  TES  and  TEMP 
•Test  Strategy  Coordinated  and  Optimized  with  ITT 

•Major  Range  Coordination  /  Long  Lead  Planning  Requirements  Established 
•Integrated  T&E  Strategy  and  Approach  Addressing  the  Total  Program  Lifecycle 
•Event  Driven,  Measurable  Modeled  Test  Approach  Logically  Sequenced  and  Optimized 
•Operationally  Realistic  Environments  and  Measurements  Defined 
•Verifiable  Requirements  with  Established  Completion  Criteria 

•Unique  Test  Requirements  Needed  to  Complete  the  System  Design  Requirement  Set 
•Integrated  Test  Plan  aligned  with  Program  Risk  Plan 

•Early  Test  Plans  and  Facilities  Definitions. 

•Requirements  Based  on  Tested  Architecture 
•Streamlined  Testable  Architecture 
•Event  Based  Test  Schedule  Developed 
•Integrated  Risk  Program 

•Scientific  and  Statistical  methods  applied  to  Test  Plan 
•Verifiable  Requirements  &Verification  Statements  Developed 
•Test  Unique  Design  Requirements  Developed. 

•Embedded  Operational  Realism  in  Test  Program 
•Support  to  Operational  Sustainment  Assessment  As  Required 


TV&P  Ensure  Intent  is  Captured  in  the  Verification  Program 
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Conclusions 


NORTHROP  GRUMMAN 


•  Maintaining  the  original  intent  of  the  product  is  mandatory  for  the  Warfighter's  success. 

•  Intent  is  sourced  from  multiple  places 

-  Mission  Statement,  TES  ,  TEMP,  OPSCON  /  CDD 


•  Attention  to  the  intent  must  be  maintained  throughout  the  program's  lifecycle 


•  Avoid  the  temptation  to  complete  requirements  verification  to  the 
remember  how  the  product  must  perform  the  requirements. 

•  Tools  and  processes  exist  today  to  help  avoid  these  pitfalls. 

-  Modeled  Test  Plan 

-  Modeled  Tests 

-  Coordinated  Working  Groups 


"Letter  of  the  Spec" 


-  Data  Plan  supporting  a  Program's  Lifecycle  including  early  will  help  offset  total 
program  costs 


•  Too  many  programs  are  driven  by  cost  and  schedule  at  the  expense  of  performance 

-  PMs  must  embrace  the  idea  of  Integrated  Testing 

-  Ensure  Programs  start  with  the  proper  skill  mix 
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Questions 


NORTHROP  GRUMMAN 
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Contact  Information 


NORTHROP  GRUMMAN 


Steve  Scukanec 
"The  Test  Guy" 

Flight  Test  and  Evaluation 
Aerospace  Systems 
Northrop  Grumman  Corporation 
Stephen.Scukanec@NGC.com 
310-350-3156 
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NORTHROP  GRUMMAN 


A  Day  in  the  Life  of  a  Verification 

Requirement-  Tutorial 

27th  Annual  National  T&E  Conference 

Marriott  Tampa  Waterside 

March  14th,  2011 


Stephen  Scukanec 

Senior  Test  Engineering 
Flight  Test  and  Evaluation 
Northrop  Grumman 
Aerospace  Systems 
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Verification  Requirements  -  What  Are 
They  And  Why  Do  We  Need  Them? 


NORTMROP  GRUMMAN 


•  Verification  requirements  specify  the  verification  events 
needed  to  prove  the  satisfaction  of  the  product 
requirements  and  help  to  define  the  verification  process 
and  environment 


•  Verification  requirements  are  necessary  for  at  least  two  ^ 
reasons: 

-Existence  of  verification  requirements  demonstrates 
verifiability  of  product  requirements 

-  Agreed-to  verification  requirements  define  the 
verification  program  by  which  the  contractor  shows 
that  the  product  is  what  the  customer  contracted  for 
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A  Day  in  the 
Requirement 


of  a  Verification 


NORTMROP  GRUMMAN 


Product  Requirements 


Design  Specification  Section  4  Charactistics 


H - 

m  !H 

IPT  - 

[ 

Establish 

Establish 

Establish 

Develop 

•Deyejdp -Detailed  • 

Proof  of  Design 

Certification  /  SOF 

Acceptance 

Verification  Matrix 

: : ;  Verification: ; : ; 

Verification 

Verification 

Verification 

(VCRI/VCRM) 

: :  R^quirerriertts:  • : 

Statements  (A) 

Requirements  (B) 

Requirements  (C) 

(D) 

- 1  IPT  /  Ml 

h 

-  IPT/V&V 

-  IPT/V&V 

IPT/V&V 

These  Documents  May  Be 
Combined  Depending  on 
Program  Direction  or  Product 
Requirements 


IPT  |  Develop  Product 

Design  Verification 
Plan 
(F) 


Develop  Test  / 

Develop  Modeling 

Demonstration 

and  Simulation 

Plan 

Plan 

Develop  Analysis 
Plan 


Develop 
Inspection  Plan 


Verification  events  satisfy 
the  verification 
requirements,  NOT  the 
product  requirements. 

Product  requirements  are 
never  complete  until  the 
associated  verification 


■Yes 


Perform  Data 
Analysis 


requirements  are 
completed 

The  culmination  of  the 
verification  activity  of  the 
design  requirements 
results  in  a  verified 
product. 


Ye; 

H 


Report 


Archive  Data 
Package  With 
Configuration 
Management 


Submit  Package 
To  Customer 


Finish 


) 


{>  DD-250 
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Start  with  Product  Requirements 


NORTMROP  GRUMMAN 


•  The  verification  process  begins  with  authenticated  product  requirements 

•  Examples: 

-  PR-1  :LRU  markings* 

•  The  product  line-replaceable  units  shall  be  marked  in  accordance  with  MIL-STD- 
130M. 

-  PR-2:  Operational  availability 

•  The  product  shall  have  an  operational  availability  (A0)  of  97.5%  at  IOC. 

-  PR-3:  Flight  performance 

•  The  Transportation  Management  Center  shall  handle  up  to  15  major  incidents  and  30 
minor  incidents  during  peak  travel  hours. 

-  PR-4:  LRU  accessibility* 

•  Each  product  line-replaceable  unit  shall  be  able  to  be  removed  and  replaced 
without  removing  any  other  item  or  displacing  any  cables. 

-  PR-5:Recovery  force  communication  -  nominal 

•  The  product  shall  provide  a  communications  system  capable  of  communicating 
with  the  recovery  forces  pre-  and  post-  landing 


Verify  aN  product  requirements,  not  just  functional/performance  requirements 
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Create  Verification  Criteria 


NORTHROP  GRUMMAN 


Design 
Requirement 

V 


3.2.2.15.34  Recovery  Force  Communications 
The  product  shall  provide  a  communications  system  capable  of 
►  communicating  with  the  recovery  forces  pre-  and  post-  landing 


Verification  Objective 


Design 

Verification  / 


Perform  Integrated  System 
Test  of  the  communications 
system  capability  to  provide 
a  voice  communications  and 
.  beacon  with  recovery  forces 
J\  pre  and  post  landing  within 
V an  integrated  hardware  / 
/software  environment 


v 


Perform  a  demonstration  of 
the  communications  systems 
capability  to  provide  voice 
and  beacon  communications 
with  recovery  forces  pre  and 
post  landing  while  within  a 
representative  environment 
and  using  a  production 
equipment  configuration 


Pass  /  Fail  (Success  Criteria) 


Testing  will  show  that  the 
communications  system  can 
transmit  and  receive  audio 
at  frequencies  and  ranges 
(power)  represented  by 
standard  ground  recovery 
force  communications 
devices  as  defined  in  TBD 


Demonstration  will  show  the 
ability  for  the 

communications  systems  to 
verbally  communicate  with 
the  on-board  communication 
production  configuration 
equipment.  The 
demonstration  will  also 
show  beacon  tracking  within 
communication  ranges 
established  by  TBD. 


Verification  Cross  Reference  Matrix 


Traceability 


Paragraph  # 

N/A 

1 

A 

M/S 

D 

T 

Y  2. 2. 15.34 

VR-5T 

^  3.2.2.15.34 

VR-5D 

SE  -  Translates  Operational 
Objectives  into  Design 
Requirements 

Design  -  Provides  assessment  of 
requirements  implementation 
Test  -  Provides  assessment  of 
requirements  verifiability 


SE  -  Provides  compliance  of  the 

design  requirement 

Test  /  Implementation  Group  - 

Ensures  Verification 
Implementation  Feasibility 
Advises  alternatives  to  support 
programmatics 
Assesses  completeness 
Provides  verifiability  assessment 


SE  -  Verification  Allocation  and 
Traceability  Assurance 
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Identifying  a  verification  method  is  necessary,  but  not  sufficient! 
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Verification  Requirement 
Attributes 


NORTHROP  GRUMMAN 


Verification 

Requirements 

•Inspection 

•Analysis 

•Modeling  and 

Simulation 

•Demonstration 

•Test 


Must  answer 
5  Questions 


^Objective 

What  is  the  purpose  of  this  verification? 

HMethod 

What  method  do  you  need  performed?  What  are 
the  verification  circumstances  (e.g.,  laboratory, 
desk-top  analysis,  flight  test)? 

^Environment 

What  are  the  environmental  conditions  under 
which  the  item  will  be  verified? 

ElSpecial  Conditions  (if  necessary) 

Are  there  any  unique  conditions  (e.g.,  item 
configurations)  necessary  for  the  execution  of 
the  verification? 

^Success  Criteria 

What  results  are  to  expected? 
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Verification  Methods 


NORTHROP  GRUMMAN 


•  Inspection: 

-  An  element  of  verification  that  is  generally  nondestructive  and  typically  includes  the 
use  of  sight,  hearing,  smell,  touch,  and  taste;  simple  physical  manipulation;  and 
mechanical  and  electrical  gauging  and  measurement.  (MIL-STD-961E;  called 
Examination) 


•  Analysis: 

-  An  element  of  verification  that  uses  established  technical  or  mathematical  models  or 
simulations,  algorithms,  charts,  graphs,  circuit  diagrams,  or  other  scientific 
principles  and  procedures  to  provide  evidence  that  stated  requirements  were  met. 
(MIL-STD-961E) 

•  Demonstration: 

-  An  element  of  verification  that  involves  the  actual  operation  of  an  item  to  provide 
evidence  that  the  required  functions  were  accomplished  under  specific  scenarios. 
The  items  may  be  instrumented  and  performance  monitored.  (MIL-STD-961E) 


Test: 


-  An  element  of  verification  in  which  scientific  principles  and  procedures  are  applied 
to  determine  the  properties  or  functional  capabilities  of  items.  (MIL-STD-961E) 


Verification  isn't  ONLY  test! 
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Sample  Verification  Requirements  - 1 


NORTHROP  GRUMMAN 


•  VR-11:  compliance  of  product  markings  shall  be  verified  by 
examination  of  design  drawings  at  the  LRU  supplier’s  location 
prior  to  the  LRU  CDR.  The  inspection  will  show  that  each 
marking  on  the  LRU  conforms  to  MIL-STD-130M. 

•  VR-2A:  the  product  operational  availability  shall  be  calculated 
using  the  results  of  the  government-accredited  contractor- 
developed  reliability  and  maintainability  analyses  performed 
during  the  design  in  conjunction  with  the  design  reference 
missions  documented  in  report  xxxx.  The  analysis  will  show  that 
the  product,  in  its  operational  environment,  supported  with  its 
support  equipment  and  personnel,  across  all  missions,  will  have 
an  operational  availability  of  at  least  97.5%. 
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Sample  Verification  Requirements  -  2 


NORTHROP  GRUMMAN 


•  VR-3MS:  Verification  of  the  TMC’s  handling  of  15  major  and  30  minor  incidents 
during  peak  hours  shall  be  shown  through  a  live  simulation.  The  TMC  training 
simulator  shall  be  configured  for  peak  a  peak-travel-hours  training  class  and 
staffed  with  trained  TMC  operators.  The  training-simulator  operator  shall  inject 
various  combinations  of  major  and  minor  incidents  over  the  peak-travel  period 
and  the  TMC  performance  shall  be  recorded  digitally  and  using  digital  cameras. 
The  simulation  shall  be  repeated  using  different  combinations  of  TMC  operators 
and  sets  of  incident  combinations.  Verification  shall  be  achieved  when  the  TMC 
handles  all  simulated  sets  of  incidents  with  all  combinations  of  operators  with  no 
equipment  or  software  overloads  or  interrupts  and  with  no  operator  overloads  or 
interrupts. 

•  VR-4D:  Removal  and  replacement  of  all  LRU’s  shall  be  demonstrated  on  the 
aircraft  to  show  that  each  LRU  can  be  removed  and  replaced  without  removing 
any  other  items  or  moving  any  cables. 

•  VR-5D:  Perform  demonstration  to  provide  a  communications  system  capable  of 
communicating  with  the  ground  command  team  while  in  a  representative 
environment  and  production  configuration.  Demonstration  will  show  capability  to 
communicate  with  recovery  forces  at  TBD  distances  in  the  TBD  terrain 
environment. 
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Sample  Verification  Requirements  -  3 


NORTHROP  GRUMMAN 


VR-5T:  Prove  that  the  product’s  communications  system  is 
capable  of  communicating  with  the  ground  command  team  by 
performing  an  integrated  system  test  within  an  integrated 
hardware/software  environment.  Testing  will  show  that  the 
product  can  transmit  and  receive  to  standard  ground  recovery 
forces  audio  at  frequencies  represented  by  communications 
devices  defined  in  (TBD). 


Verification  Objective 


Verification  Method 


Note  -  there  are  no 
Special  Conditions 


Environment 
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Verification  Requirements  Flow  and 
Traceability 


NORTHROP  GRUMMAN 


Specification 


PR-1 

PR- 2 

PR-3 

PR-4 

PRp5^ 

VR-1I 

VR-2A 

VR-3MS 

VR-4D 

VR-5D 

VR-5T 

Verification  Requirements 
Appear  in  the  Same 
Specification  as  the  Product 
Requirements  to  be  Verified 


Master  Verification  Plan 


Inspection  VR-1I 
Analysis  VR-2A 
Modeling  and 
Simulation 

VR-3MS 

Demonstration 

VR-4D,  VR-5D 

Test 

VR-5T 


Verification 

Requirements 


Product 

Requirement 

N/A 

Insp 

Anal 

M&S 

Demo 

Test 

Verification 

Requirement 

PR-1 

X 

VR-ll 

PR-2 

X 

VR-2A 

PR-3 

_ V _ 

VR-3MS 

yv 

PR-4 

X 

VR-4D 

PR-5 

X 

X 

VR-5D 

VR-5T 
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Create  Detailed  Verification 
Requirements  (Verification  Events) 


NORTHROP  GRUMMAN 


Convert  verification  statements  into 
detailed  verification  requirements 
(verification  events)  by  — 


Master  Verification 


if) 

C 


<D 


! 


ro 

u 


$ 


Plan  (MVP) 


Inspection  VR-1I 
Analysis  VR-2A 


Modeling  and 
Simulation 

VR-3MS 


Demonstration 

VR-4D,  VR-5D 
Test 
VR-5T 


O  <  Q.  < 
<  ®  ® 
fl> 


w  ®  ai  (D 

3=3 


n> 


fD 


A) 


(A  (A 

0) 

-t 

fl) 


A  One  To  One  Relationship  Exists  Between 
the  Verification  Requirements  and  the 
DVRs 


I For  each  verification  activity  identified  in  the 
verification  matrix,  develop  a  detailed 
description  of  the  activity  including: 


< 


•Verification  configuration  and  its  relationship 

to  production  configuration 

•Associated  prerequisites 

•Constraints 

•Objectives 

•Procedures 

•Relevant  environmental  conditions 
•Pass/fail  criteria-  and  necessary  Data  Set, 
•Analysis  models,  if  applicable. 

•Sequence  if  applicable 

•Verification  Environment  (e.g..  Lab,  Flight, 


^Production) 
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Master  Verification  Plan 
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Master  Verification  Plan 


Inspection  VR-1I 
Analysis  VR-2A 
Modeling  and 
Simulation 


VR-3MS  i 

Demonstration 


VR-4D,  VR-5D 

Test 


Customer 

Concurrence 


lAawly/*a#yAif 
lrc7J  illLilllUII 

Mod&lma 

mwwGBGwwww^J 
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Verification  Execution  Flow 
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_ K 


Procedures  Execute ^  RePorts 


Method 

Organization 

Early  Verification  Benefits 

Inspection 

QA,  Manufacturing, 
Mission  Assurance 

•Inspection  Points  Identified 
•Tooling  Requirements  Identified 

Analysis 

Systems  Engineering 
Specialty  Engineering 

Design 

•Define  /  Build  /  Buy  /  Train 
Analysis  Prior  to  Need  Date 

•Accreditation  of  Analyses  Tools 
Prior  to  Need  Date 

Modeling 

and 

Simulation 

Systems  Engineering 
Specialty  Engineering 

Design,  Operational 
Assessment 

•Define  /  Build  /  Buy  /  Train 
Modeling  and  Simulation  Tools 

Prior  to  Need  Date 

•Accreditation  of  Models  Prior  to 
Need  Date 

Demo  & 

Test 

Ground  and  Flight 

Test 

Facilities 

Development 

•Laboratory  and  Lab  Software 
Requirements  Identified 

•Facilities  Requirements  Identified 
•Long  Lead  Test  Items  Identified 

Early  Verification 
Supports  Multiple 
Organizational 
Functions'  Long  Lead 
Needs  and  Prevents 
Costly  Late  Program 
Re-Work 
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Planning  for  Verification  Execution  and 
Product  Verification 


Rev  1 


Define  Verification 
Requirements  Early 
and  in  Detail  to 
Establish  the  Entire 
Verification  Effort 


0 


...  and  it  Costs 
Relatively  little 


Rev  2 


Rev  3 


Rev  4 


Requirements 


Design 


Build 


Verification 


Long  Lead  Facilities 
Laboratory  Design 
Range  Coordination 
Design  Requirements 
Software 
Analysis  Tools 


Certification! 


Discover  the  Verification 
Requirements  Late  and  Have 
Enormous  Rework  to  Establish 
the  Entire  Verification  Effort 


Early  Verification  Is  an  Effective  Cost  Avoidance  Approach 
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Requirements:  The  Good  ... 
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•  The  (radio  set)  design  shall  allow  trained  operators  and  maintainers 
to  perform  all  critical  tasks  required  to  install,  operate  and  maintain 
the  (radio  set)  correctly  on  the  first  attempt  90%  of  the  time. 

•  The  XYZ  satellite  shall  be  launched  on  a  Delta  IV  EELV  or  on  an 
Atlas  V  EELV. 


•  The  XYZ  spacecraft  shall  rendezvous  with  the  ISS  in  accordance 
with  the  Interface  Definition  Document  (IDD)  for  International  Space 
Station  (ISS)  Visiting  Vehicles  (VVs),  SSP  50235  . 

•  The  XYZ  spacecraft  shall  shall  perform  the  precision  approach 
maneuver  to  the  ISS  in  accordance  with  the  Interface  Definition 
Document  (IDD)  for  International  Space  Station  (ISS)  Visiting 
Vehicles  (VVs),  SSP  50235. 

•  The  XYZ  spacecraft  shall  dock  with  the  ISS  in  accordance  with  the 
Interface  Definition  Document  (IDD)  for  International  Space  Station 
(ISS)  Visiting  Vehicles  (VVs),  SSP  50235. 
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Requirements:  ...  The  Bad  ... 
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•  The  ship  and  all  systems  shall  be  designed  to  minimize 
maintenance.  Maintenance  personnel  shall  be  provided  the 
necessary  tools,  information,  technical  documentation  and  skills  to 
perform  maintenance. 

•  The  Product  shall  provide  controls  and  displays  to  facilitate 
operator  interaction  in  carrying  out  all  assigned  missions. 

•  And,  of  course, ... 

The  Product  shall  be  user-friendly. 
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Requirements:  ...  And  The  (Truly)  Ugly. 
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•  Human  Systems  Integration  (HSI)  characteristics  and  capabilities  for 
(the  ship)  will  include  human  factors  engineering,  personnel, 
habitability,  manpower,  training,  environment,  safety  and  occupational 
health  (ESOH)  and  personnel  survivability.  HSI  processes  will  be  used 
to  maximize  human  performance  effectiveness,  reliability,  readiness 
and  safety  of  the  ship  and  crew  while  minimizing  system  life-cycle 
costs  through  iterative  analysis  and  design  tradeoffs. 

•  All  systems  shall  be  designed  for  maintainability.  Reductions  in 
manpower  requirements  for  system  maintenance  (both  planned  and 
unscheduled)  shall  be  achieved  through  an  in-depth  analysis  of 
maintenance  related  tasks,  early  identification  of  maintenance 
concepts,  and  definition  of  maintenance  requirements  and  constraints 
early  in  the  design  process.  Burdens  imposed  on  manpower, 
personnel  and  training  related  to  system  maintenance  shall  be 
identified  as  early  as  possible  and  refined  throughout  the  development 
process. 

•  The  ship  shall  be  capable  of  being  operated  and  maintained  without 
requiring  significant  new  knowledge,  skills,  abilities,  aptitudes  or 
physical  characteristics  of  the  core  crew  and  mission  package  crews. 
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Requirement  Generation 
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•  Class  Exercise  -  Generate  a  good  requirement  as  agreed  to  by  the  team  and 
then  let’s  test  the  theory 

-  Generate  a  Requirement  for  the  following  Methods 

•  Analysis 

•  Test 

•  Inspection 


Layer  1 


nr\ 


□ 
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What  is  Verification  ? 
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What  is  Verification? 


NORTMROP  GRUMMAN 


•  Confirmation,  through  the  provision  of  acceptable  objective  evidence,  that 
specified  requirements  have  been  fulfilled.  (MIL-STD-961E) 


Disabled  but  enabled  in  current  group  (groupl) 


Verification  Manager  —  groupl 

test_signals 

□  Check  Static  Upper  Bound  1 

. 0 

. □  Check  St*t«  Upper  Bound  3 

0  Check  5iat«  Upper  Bound  4 

EJ  fej  Subsystem 

L . □  Check  Static  Upper  Bound  5 
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Verification  Requirement 
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•  A  Verification  Requirement's  Purpose 


-  Establishes  the  Requirements  Intent 

•  “If  the  Unit-Under-Verification  (UUV)  performs  this 
way  (or  has  characteristics),  it  is  compliant  with  the 
requirement” 

-  Establishes  the  completion  criteria 

•  “Requirement  contract  with  the  customer” 
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Verification  Requirement 
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•  Verification  Requirement  Levels 

-  Developed  for  all  specification  levels  (where  ever  there  is  a  shall) 

-  Written  at  the  level  of  the  unit  configuration  defined  by  the  specification  configuration 
item 

•  “The  Item  Under  Verification  is  the  Title  of  the  specification” 

•  If  you  can’t  generate  a  Verification  Requirement  at  the  level  of  the  specification- 
defined  item,  then  the  requirement  is  written  at  the  wrong  level” 


ol  ^ 


oHol  <C 


ol  ol  ol 


ol  ol  ol  ol 
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Verification  Requirement 
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•  A  one  to  many  relationship  exists  between  the  “Shall”  and  the  associated 
verification  requirement's). 

-  Example 

•  The  Product  shall  provide  visual  operator  status  of  power  to  the  unit  when  applied  for 
both  proper  and  improper  power  conditions  established  by  ICD  XXX. 

•  Demonstration  1 

-  Verify  by  Demonstration  that  when  power  is  supplied  to  the  unit  in  accordance 
with  the  established  interface  defined  in  ICD  XXX  the  operator  is  provided  visual 
status.  Demonstration  will  show  that  when  proper  power  is  provided  a  continuous 
visual  indication  is  provided  to  the  operator. 

•  Demonstration  2 

-  Verify  by  Demonstration  that  when  power  is  supplied  to  the  unit  outside  of  the 
limits  established  by  ICD  XXX  the  operator  is  provided  a  visual  cue  different  than 
the  nominal  power  indication.  Demonstration  will  show  that  when  invalid  power  is 
provided  a  visual  indication  of  improper  power  is  provided  to  the  operator  for  as 
long  as  the  improper  power  conditions  exist. 


•  Note  -  this  verification  statement  should  alert  the  requirements  team  that  a 
requirement  for  the  unit  to  prevent  unit  damage  in  the  event  of  improper  power 
application  should  have  been  written.  Another  advantage  of  developing  the 
Verification  Requirement  early  Forgotten  Requirements  -  helps  to  identify  missing 
Product  Requirements. 
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Verification  Requirements 
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•  Key  points  to  writing  a  good  verification  requirement 

-  Verify  by  (insert  method  here)  that . 

-  The  (use  above  method)  will  show  that . 

•  Ability  to  create  a  verification  requirement  ensures  that  the  Product  requirement  is 
verifiable 
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Verification  Requirement  Attributes 
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Verification 

Requirements 

•  Inspection 

•  Analysis 

•  Modeling  and 
Simulation 

•  Demonstration 

•  Test 


Must  answer 
5  Questions 


^Objective 

What  is  the  purpose  of  this  verification? 

HMethod 

What  method  do  you  need  performed?  What  are 
the  verification  circumstances  (e.g.,  laboratory, 
desk-top  analysis,  flight  test)? 

^Environment 

What  are  the  environmental  conditions  under 
which  the  item  will  be  verified? 

ElSpecial  Conditions  (if  necessary) 

Are  there  any  unique  conditions  (e.g.,  item 
configurations)  necessary  for  the  execution  of 
the  verification? 

^Success  Criteria 

What  results  are  to  expected? 
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What  is  Verification  ? 
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Class  exercise 
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•  Break  into  teams 

-  4  people  per  team 

•  Product  Requirement  Owner  (IPT,  SEIT,  Other) 

•  Designer  (Hardware,  Software) 

•  Verification  Execution  Representative  (l&T,  Q.A.,  Analyst  etc;) 

•  Verification  Team 
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Class  Exercise 
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•  Product  Requirement  Owner 

-  Establishes  the  requirement  intent  with  a  verification  requirement 

-  Creates  the  initial  VR 

•  Design  Team 

-  Agrees  that  the  design  is  capable  of  performing  the  success  criteria 

•  Verification  Team 

-  Ensures  depth  and  breadth  of  the  requirements  are  met  with  the 
success  criteria  (nominal  /  off  nominal,  needed  analysis,  modeling 
and  simulation  techniques) 

•  Verification  Execution  team  (T&E,  Analysis  Group,  QA  etc;) 

-  Can  the  verification  requirement  be  completed? 


-  Is  it  cost  effective  ? 
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Verification  Requirement  Evaluation 
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•  Evaluate  the  generated  Product  requirement  and  verification 
requirement(s)  set 


-  Did  you  have  to  re-write  your  Product  requirement? 


-  Did  the  Product  requirement  provide  data  to  establish  the  success  criteria? 


-  Is  the  specified  verification  environment  consistent  with  the  operational  objectives  as 
established  in  the  Product  requirement? 


Report  Card 

5U  BJCCT 

GRADE 

Advancing  the  State  of  the 
Window*  programming  art 

B” 

Software  deployment 

A- 

Security 

B- 

Richness,  reach,  and 
ne^t-generation  Ut 

inoon^tets 

Web  services 

A 
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Verification  Benefits 
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Examine  Programmatic  Benefits 
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•  The  Customer  Benefits  ?? 

-  Knows  how  the  Product  requirements  will  be  satisfied  from  the  beginning 

•  Cost  Benefits  ?? 

-  Better  Cost  Estimates.  You  know  what  you  need  to  do. 

•  Schedule  Benefits  ?? 

-  Better  Schedule  Estimates.  You  can  scope  the  entire  task  early  providing  a 
better  schedule 


•  The  PMO  Benefits  ?? 

-  Knows  what  the  needs  are  to  prove  satisfaction  of  the  Product  requirement. 
Knows  what  “Customer  Satisfaction”  means  at  the  start  of  the  program. 


Better  Understanding  of  program  change 
-  Establish  impact  of  change  early 
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Early  Verification  Benefits  Examples 


NORTHROP  GRUMMAN 


Method 

Customer  / 
Organization 

Early  Verification  Benefits 

Inspection 

QA,  Manufacturing, 
Mission  Assurance 

•Inspection  Points  Identified 
•Tooling  Requirements  Identified 

Analysis 

Systems  Engineering 
Specialty  Engineering 

Design 

•Define  /  Build  /  Buy  /  Train  Analysis 
Prior  to  Need  Date 

•Accreditation  of  Analyses  Tools  Prior 
to  Need  Date 

Modeling  and 
Simulation 

Systems  Engineering 
Specialty  Engineering 

Design,  Operational 
Assessment 

•Define  /  Build  /  Buy  /  Train  Modeling 
and  Simulation  Tools  Prior  to  Need 

Date 

•Accreditation  of  Models  Prior  to  Need 
Date 

Demonstration 
and  Test 

Ground  and  Flight  Test 
Facilities  Development 

•Laboratory  and  Lab  Software 
Requirements  Identified 

•Facilities  Requirements  Identified 
•Long  Lead  Test  Items  Identified 
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Verification  Modeling 
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Verification  Modeling 
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Parent-Child  Relationship 

-  Method 

-  Environment 

-  Success  Criteria 

-  The  Verification  Pyramid 

•  Verify  at  the  lowest  level 

•  Verify  Once 

•  Verify  under  operational 


■Aa rific&Ron 

w&rnTG&aTwWr 

NiOi 

WTwWf&trrT^P 

environmental  conditions 


That  Really  Means 


You  only  conduct  environmental  qualification  on  a  UUV  one  time 

at  the  Box  Specification  level 
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Verification  Modeling 
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Verification  Requirements  Key  Attributes 
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Verification  Modeling  Example 
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S- 


Verification  Requirements  Key  Attributes 


Method 


(t) 


Method 


(t) 


Method 


(t) 


i 

f 


Environment 

(100  Hours) 


Success  Criteria 

(+-  2  feet) 


Contract 

Specification 


*1 

+- 


Environment 

(400  Hours) 


A 

J  4. 


A 


Success  Criteria 

(+-  20  feet) 


Product 

Specification 


+ 


Environment 


Success  Criteria 


1 - 

-  * - 1 

-  * - 1 

)  1 

/  ® 

o 

j 


Prime  Item 

Development 

Specification 


Mil 


o 

o 

o 

o 


Looking  at  the  verification  requirement  flow  down  ensures 
A  Consistent  cost  effective  verification  program 
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Verification  Modeling  -  A  Closer  Look 
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Verification  Requirements  Key  Attributes 


Contract 

Specification 


Product 

Specification 


Prime  Item 

Development 

Specification 

O 
O 


Is  400  too  much  at  thiSy/evel?  Is  20^eet  over-constrainii^ 
Is  1000  too  much  at  this  level?  Is  200  feet  over-constraining 

testing  *  $$  0^,  Design^ 1 


Over 
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Verification  Modeling  Another 
Example 
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Verification  Requirements  Key  Attributes 


\ 

Method  (t) 

\ 

/ 

Environment 

(100  Hours) 


Success  Criteria 

(+-  2  feet) 


nroi 


Environment 

(100  Hours) 


Success  Criteria 

(+-  2  feet) 


Contract 

Specification 


Prime  Item 

Development 

Specification 


O 

O 

o 

o 


39 


Cleared  for  Public  Release,  Control  No.  08-105  Steve  Scukanec  Northrop  Grumman  3/14/2011 


Modeling  Tools 
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•  Several  Standard  tools  exist  to  conduct  the  modeling  activities: 
-  DOORS 
-CORE 


-  Excel  (doable  but  it’s  the  hard  way) 


-  Others? 
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Verification  Modeling 
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•  A  technique  to 

-  Establish  verification  metrics 

-  Track  the  verification  program 

-  Track  risk  activities  assigned  to  the  verification  program 

-  Ensure  proper  verification  flow  down 

-  Ensure  operational  environment  properly  flowed 

-  Help  determine  lost  requirements 

-  Help  track  design  functions 

-  Assist  in  verification  program  prioritization 

•  Can  be  connected  to  requirements  traceability  tools 

•  Allows  for  easier  design  completeness  assessments 
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The  Verification  Cross  Reference  Index  (VCRI) 
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•  Why  A  VCRI* 

-  A  tool  for  tracking  requirements  compliance 

-  A  tool  to  quickly  assess  that  at  least  one  verification  condition  exists  for 
each  requirement 

•  What  a  VCRI  is  not 

*  Not  the  verification  requirement  set 

*  Not  the  definition  of  the  verification  requirements 

•  Some  Conclusions 

*  The  VCRI  results  from  the  development  of  the  Verification 
Requirement 

*  Having  only  a  VCRI  can  develops  “Bad  Habits” 

*  Adds  no  value  without  the  Verification  Requirement 


*Also  known  as  the  Verification  Cross  Reference  Matrix  (VCRM) 
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Create  Verification  Cross-Reference 
Matrix 


3.2.2.15.34  Recovery  Force  Communications 

The  product  shall  provide  a  communications  system  capable  of 

communicating  with  the  recovery  forces  pre-  and  post-  landing 


J\ 


Verification  Objective 


Perform  Integrated  System 
Test  of  the  communications 
system  capability  to  provide 
a  voice  communications  and 
beacon  with  recovery  forces 
pre  and  post  landing  within 
\an  integrated  hardware  / 


Design  x  „ 

Verification  /software  environment 


V 


Perform  a  demonstration  of 
the  communications  systems 
capability  to  provide  voice 
and  beacon  communications 
with  recovery  forces  pre  and 
post  landing  while  within  a 
representative  environment 
and  using  a  production 
equipment  configuration 


Pass  /  Fail  (Success  Criteria) 


Testing  will  show  that  the 
communications  system  can 
transmit  and  receive  audio 
at  frequencies  and  ranges 
(power)  represented  by 
standard  ground  recovery 
force  communications 
devices  as  defined  in  TBD 


Demonstration  will  show  the 
ability  for  the 

communications  systems  to 
verbally  communicate  with 
the  on-board  communication 
production  configuration 
equipment.  The 
demonstration  will  also 
show  beacon  tracking  within 
communication  ranges 
established  by  TBD. 


Verification  Cross  Reference  Matrix 


Traceability 


Paragraph  # 

N/A 

1 

A 

M/S 

D 

T 

Y  2. 2. 15.34 

VR-5T 

^  3.2.2.15.34 

VR-5D 

SE  -  Translates  Operational 
Objectives  into  Design 
Requirements 

Design  -  Provides  assessment  of 
requirements  implementation 
Test  -  Provides  assessment  of 
requirements  verifiability 


SE  -  Provides  compliance  of  the 

design  requirement 

Test  /  Implementation  Group  - 

Ensures  Verification 
Implementation  Feasibility 
Advises  alternatives  to  support 
programmatics 
Assesses  completeness 
Provides  verifiability  assessment 


SE  -  Verification  Allocation  and 
Traceability  Assurance 


Identifying  a  verification  method  is  necessary,  but  not  sufficient! 
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Verification  Program  Events 


NORTMROP  GRUMMAN 


•  Pre-Contract  Award 

-  Establish  operational  environment  construct 

•  Use  system  specification  and  develop  verification  criteria 

-Customer  Should  Pass  operational  environment  to  the 
contractor 

•  Contract  Award 

-  Establish  Project  Specification  Verification  Statements  and  get 
customer  concurrence 

•  At  Specification  Requirements  Review  (SRR)  /  Subordinate  reviews 

-  Determine  Requirements  Verifiability 


Verification  starts  when  the  program  starts 
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Planning  for  Verification  Execution  and 
Product  Verification 


NORTHROP  GRUMMAN 


Review  1 


Review  2 


Review  3 


Review  4 


Capture 

Operational 

Environment 


Define  Verification 
Requirements  Early 
and  in  Detail  to 
Establish  the  Entire 
Verification  Effort 


0 


...  and  it  Costs 
Relatively  little 


Long  Lead  Facilities 
Laboratory  Design 
Range  Coordination 
Design  Requirements 
Software 
Analysis  Tools 


Requirem 

ints 

Design 

B 

uild 

Verifica 

tion 

V 

Certification! 

Discover  the  Verification 
Requirements  Late  and  Have 
Enormous  Rework  to  Establish 
the  Entire  Verification  Effort 


Early  Verification  Is  an  Effective  Cost  Avoidance  Approach 
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Summary 


NORTHROP  GRUMMAN 


•  Early  development  of  verification  requirements  helps  develop  good  product 
requirements 

•  Early  development  of  verification  requirements  helps  identify  missing  requirements 

•  Verification  is  the  communications  link  to  the  design  and  execution  teams 

•  Verification  customers  are  across  the  entire  program 

•  Verification  identifies  when  the  design  is  complete 

•  Early  development  of  verification  requirements  can  ensure  the  operational  environment 
is  captured  across  the  test  /  demonstration  program 

•  The  VCRI  /  VCRM  is  necessary,  but  not  sufficient,  for  verification 

•  Verification  modeling  helps  develop  a  “one  time  only”  verification  program 

•  Verification  increases  the  Program’s  cost  effectiveness 
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Contact  Information 


NORTMROP  GRUMMAN 


•  Stephen  Scukanec 
“The  Test  Guy” 

Northrop  Grumman 
Stephen.Scukanec@ngc.com 

•  310-350-3156 
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DPG  as  the  Chem/Bio  MRTFB  Activity 

Jean  Baker 

16  March  2011 


Major  Range  and  Test  Facility  Base 

(MRTFB) 


•1971:  DoD  recognizes  that  large  military  test 
facilities  represent  national  assets  and 
establishes  MRTFB 

•Set  of  test  installations,  facilities  and  ranges 
•Selected  for  unique  T&E  assets 
•Includes  installations  from  the  Services,  Joint 
Interoperability  Test  Command  and  range 
cooperatives 
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MRTFB  Activities 


Cold  Regions  Test 
Center 


Tropic  Regions  Test 
Center 


Dugway  Proving  Ground 
(West  Desert  Test  Center) 


Keyport  Pacific  Northwest 
Range  Complex 


Nevada  Test  and  Training 
Range 


Naval  Air  Warfare  Center  -Weapons 
Division,  China  Lake 


30th  Space  Wing 


Air  Force  Flight  Test 
Center 


Yuma  Test  Center 


Electronic  Proving 
Ground 


Utah^Fe^t  and  Training 
Range 


48th  Test  Group 


Naval  Air  Warfare  Center  -Weapons 
Division,  Point  Mugu 


46th  Test  Wing 


Pacific  Missile  Range 
Facility 


White  Sands  Test 
Center 


High  Energy  Laser  Systems 
Test  Facility 


Joint  Interoperability  Test 
Command 


Aberdeen  Test 
Center 


Naval  Air  Warfare  Center- 
Aircraft  Division, 
Patuxent  River 


Eagle  Lab  Testing  Center 


45th  Space  Wing 


Atlantic  Undersea  Test  and 
Evaluation  Center 


Kwajalein  Missile  Range 


''Defense  Information  Systems  Agency 
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Dugway  Becomes  Part  of  the  MRTFB 


Attack  on  Pearl  Harbor  prompted  the  formation  of 
Dugway  Proving  Ground  (DPG)  by  President  Roosevelt  in 
1942 

Established  to  fill  the  need  for  supporting  the  Chemical 
Warfare  program  in  WWII 
Became  part  of  the  MRTFB  in  1971 
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West  Desert  Test  Center  (WDTC) 

•  WDTC  is  the  mission  side  of  DPG 

•  Primary  mission:  testing  chemical  and  biological  (CB) 
defense  systems 

•  Perform  nuclear,  biological,  and  chemical  (NBC) 


contamination  survivability  testing  of  defense  materiel 

•  State-of-the-art  laboratories  and  chambers 

•  Extensive  field  testing  grids 
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Terrain 


798,214  acres  of  Great  Basin  desert  terrain  ranging  from 
salt  flats,  to  intermittent  sand  dunes  and  rugged  mountains 
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Airspace 


Adjacent  U.S.  Air  Force  gunnery  and  bombing  ranges 
extend  Dugway’s  restricted  airspace  to  an  area  of  about 
90  x  70  miles  and  up  to  an  elevation  of  58,000  feet 
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Test  Grids,  Test  Grids,  Test  Grids 


Test  grid  activities: 

•  CB  detection  systems 

•  Individual  protection 

•  Collective  protection 

•  Munitions 

•  Training 
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Field  Testing 


•  CB  simulant  dissemination  systems 

•  Referee  instrumentation 

•  WiFi 

•  Management  system  to  acquire, 
integrate,  analyze,  fuse,  and 
visualize  data  during  testing 


Smoke  and  obscurants 
Data  network 
Fiber  optic  transmission  of 
real-time  test  data 
Safari  capability 


Dissemination 


•  Vapor 

•  Explosive 

•  Powder 

•  Liquid 

•  Aerosol 

•  Aerial 
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Chemical  Testing 


•  Collective  Protection  •  Contamination  Avoidance 

•  Individual  Protection  •  Decontamination 
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Collective  Protection 


•  Chemical  warfare  agents  (CWAs) 

•  Toxic  industrial  chemicals 

•  Battlefield  contaminants 

•  Barrier  materials  swatch  testing 

•  Component  testing  of  closures  and 
seams 

•  Air  filter  component  and  air  filtration 
system  testing 

•  Full-system  tests  in  environmentally- 
controlled  chambers  using  agents  and 
simulants 

•  Outdoor  field  testing  of  full  systems  using 
CWA  simulants 


Individual  Protection 


Component  Testing 

•  Masks 


Swatch  testing 


Contamination  Avoidance 


Standoff  detection  systems 


Point  detection  systems 


Whole  system  testing 
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Decontamination 


Decontamination  Test  Facility 

•  Will  hold  all  Army  tactical  vehicles  up  to  an 
M1A1  tank 

•  Allows  for  dissemination  of  any  chemical  or 
biological  simulant 

•  Capable  of  using  any  current  known 
decontaminant  or  decontamination  system 


Small  Item  Decontamination  Test 

Fixture 

•  Any  chemical  warfare  agents 

•  Environmentally  controlled  (60  to 
110°F  and  10  to  80%  RH) 

•  Will  test  actual  fielded  small 
equipment  items  (23”x23”x36”) 

•  Biosafety  Level  (BSL)  1  biological 
simulants 
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Biological  Testing 


Aerosol  Simulant  Exposure  Chambers- 

Stainless  steel-lined  chamber  designed  to 
dispense  and  contain  aerosol  simulant  clouds  in  a 
BSL-1  and  BSL-2  environment  under  laboratory 
controls.  Primarily  used  fortesting  biological  point 
detectors,  but  can  be  used  for  any  test  requiring 
the  control  of  factors  that  could  not  otherwise  be 
controlled  in  another  setting. 


Containment  Aerosol  Chamber 

One-of-a-kind  BSL-3  chamber  designed 
fortesting  biological  aerosol  detector 
systems  by  challenging  the  systems  with 
aerosols  generated  inside  the  chamber. 


Biological  Testing,  cont’d 


Ambient  Breeze  Tunnel 

BSL-1 

Biological  point  and  standoff  detection  systems 
Referee  equipment 

Variety  of  disseminators  (Micronair  sprayers,  Skilblowers™) 
Simulants 

Interferents  (smoke,  burning  brush,  burning  tires,  burning  diesel  fuel) 
Agent-like  organisms 
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Materiel  Test  Facility 


•  Large,  environmentally-controlled 
chambers 

•  Simulant  and  live  agent  capable 

•  Point  and  standoff  detector  testing 

•  Large  equipment  testing 
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Joint  Ambient  Breeze  Tunnel 
Active  Standoff  Chamber 


Active  Standoff  Chamber 

•  Static  cloud  conditions 

•  Calibrate  referee  instruments 

•  Evaluate  standoff  CB  agent  detectors 

•  Built-in  dissemination  system 

•  Aperture  dimensions  permit  evaluation 
of  CB  detectors  at  ranges  up  to  3  km 

•  Containment  system  prevents  simulant 
leakage 


Joint  Ambient  Breeze  Tunnel 

•  Dynamic  conditions 

•  Calibrate  referee  instruments 

•  Evaluate  CB  agent  detectors 

•  Variable-pitch  fans  create  airflow 
up  to  6  meters/second. 

•  Disseminate  CB  simulants 

•  Evaluation  of  standoff  detectors  at 
ranges  up  to  3  km 


Michael  Army  Air  Field 


•  1 1 ,000-foot  runway  for  departures 

•  10,000  feet  for  landings 

•  9,000-foot  taxi  way 

•  20,000-square-foot  hangar 

•  Flight  operations  and  ground  support 
personnel 


T-'r"! 


Future  Capabilities 


Dynamic  Test  Chamber 

Designed  to  test  chemical  agent  point 
detectors  operating  in  several  MIL-STD- 
810  environmental  conditions. 


Individual  Protection  Mannequin  System 

Robotic  mannequin  designed  to  simulate  soldier 
activity  in  agent  test  facilities  in  order  to  evaluate 
individual  protection  ensemble  performance. 


Whole  System  Live  Agent  Test  Facility 

A  BSL-3  chamber  large  enough  to  dynamically  test 
two  side-by-side  Joint  Biological  Point  Detection 
Systems  (JBPDS). 


WDTC  Capabilities  Report 
WDTC  Annual  Technology  Report 


Capabilities  Report 

2010 

West  Desert  Test  Center 


https://www.kc.army.mil/wiki/Dugway_Proving_Ground# 

Dugway_Capabilities_Report 


Annual  Technology 
Report-  Fiscal  Year  2010 


https://www.kc.army.mil/wiki/Dugway_Proving_Ground 


West  Desert  Test  Center 


Test  Technology  Division 


Report  On  Chemical/Biological  Defense  Test 
Capability  Development  at  DPG 


West  Desert  Test  Center 


US  Army,  Dugway  Proving  Ground 
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DPG  Tours 


aUe  r 

jeoo.bokei,@us.oi,mfl.iml 

0ugw«jf  proving  round 

45f-$5f-70*9 
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Assessing  System  Reliability 
Growth  When  Failure  Modes  Are 

Masked 


Patricia  A.  Jacobs 
831  656  2258  paiacobs@nps.edu 
Donald  P.  Gaver 
831  656  2605  dgaver@nps.edu 
Operations  Research  Department 
Naval  Postgraduate  School 


Series  System 


Stage  1  Stage  2 


Stage  3 


Stage  4 


•  Random  Number  of  Design  Faults/Failure  Modes  (FMs)  in 
Each  Stage/Interface 

•  When  Stage  is  Accessed  Each  Remaining  FM  May  Activate 
Independently  of  Other  FMs  with  Probabilities  Different  for 
Each  Stage 


Failure  Modes  (FMs)  and  Masking 


Each  Stage  may  contain  FMs 

If  at  least  one  FM  activates  in  stage  s  then  test 
does  not  proceed  to  stages  s+l,s+2,...,S 
-The  FMs  in  subsequent  stages  are  MASKED 

All  activated  FMs  removed  prior  to  next  test 


IF  No  FMs  Activate  In  Stage  1  & 

At  Least  One  FM  Activates  in  Stage  2 
Stages  3  &  4  Are  Masked 


Stage  1 
No  FMs 
Activate 


Stage  2 
At  least 
One  FM 
activates 


MASKED 


MASKED 


W(t;s)=Number  of  Times  Stage  s  is 
Accessed  During  Tests  1,2,..., t 

If  at  least  one  FM  activates  in  stages  l,2,...,s- 
during  test  t+1,  (Stages  s,  s+l,...,S  MASKED) 

W(t+l;s)=W(t;s) 

If  no  FMs  activate  in  stages  l,2,...,s-l  during 
test  t+1  (Stage  s  Accessed) 

W(t+l;s)=W(t;s)+l 


Model  for  Number  of  Failure  Modes 

(FMs) 

Poisson  number  FMs,  each  Stage,  prior  to  testing 

-  m(0;s)=  mean  number  FMs,  stage  s 

FM,  stage  s,  activates  with  probability  p(s) 
independently  of  other  FMs 

-  No  masking  of  FMs  within  a  stage 

If  at  least  one  FM  activates  in  stage  s  then  test 
does  not  proceed  to  stages  s+l,s+2,...,S 

-  FMs  in  subsequent  stages  MASKED 
Activated  FMs  removed  prior  to  next  test 

-  (To  be  generalized) 


Distribution  of  FMs  Remaining 


Conditional  distribution  of  number  of  FMs 
remaining  in  stage  s  after  accessed  W(t;s)  times 

Poisson  with  mean  m(0;s)(l-p(s))  /(t;s). 

Conditional  probability  0  FMs  activate  in  stage  s 
after  accessed  W(t;s)  times 

Exp{-m(0;s)(l-p(s))  /(t;s)p(s)} 

Independence  within/between  tests  strongly 
assumed 

-  No  common  cause  or  shocks  (Later!) 


Conditional  Probability  Stage  s 
Accessed  on  Test  t+1  Given 

W(t;l),...,W(t;s-l) 

•  Probability  0  FMs  Activate  in  stages  l,2,...,s-l 

a(t;s-l)=Exp{-[A(t;l)+A(t;2)+...+A(t;s-l)]} 

where 

A(t;k)=m(0;k)p(k)(l-p(k))w(t;k) 


k=l,2. 
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Simulation 
for  Test  t+1 

For  each  test  t+1  generate  a  uniform  random  variable  on  [0,1]:  U 

\J1  <  a(t;s-l)  &  U1>a(t;s) 

0  FMs  activate  in  Stages  l,2,...,s-l  &  at  least  one  s-stage-FM 
activates  on  (t+l)th  test 

Stages  s+l,...,S  MASKED 

If  0  FMs  activate  in  Stages  1,2,...,S-1,  generate  a  uniform  random 
variable  on  [0,1]:  U2 

U2<  Exp{-m(0;S)(l-p(S))w(t;S)p(S)} 

0  FMs  are  activated  in  the  last  stage,  S,  or  before 

& 

0  FMs  are  activated  in  the  entire  system  on  test  t+1 
(Optional:  another  test) 


Stopping  Rules 

•  Test:  until  0  FMs  activate,  all  stages,  R  tests 

•  Test:  until  0  FMs  activate,  all  stages,  R  consecutive 
tests 

•  Fixed  N  u  m  be  r  of  Tests 


Common  Simulation  Replication,  Number  Times 
Each  Stage  is  Accessed  &  Number  Times  0  FMs 
Activate,  All  Stages 
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Cumulative  Number  of  Times  Each  Stage  is  Accessed 
During  t  tests 
4-Stage  System: 

Mean  Initial  Number  FMs  Each  Staged 
Probability  FM  Activation=5/6 
One  Replication 


NumberofTests 


Approximate  Pooled  1-Stage  System 

•  Initial  number  FMs  has  Poisson  distribution 
with  mean  the  sum  of  the  mean  FMs  in  each 
stage 

•  Probability  a  FM  activates  =p 
—  p=sum(m(0;s)p(s))/sum(m(0;s)) 

•  Each  Test:  All  remaining  FMs  are  subject  to 
activation  (NO  MASKING) 
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If  All  Accessed  FMs  have  Same 
Activation  Probability  in  Both  S-Stage 

and  Pooled  Systems 


S-Stage  System  (MASKING) 


Number  of  tests  until  meet  Stochastically 
stopping  criterion  ^ 


Pooled  1-Stage  System 
(NO  MASKING) 

OPTIMISTIC 


Number  of  tests  until  meet 
stopping  criterion 


Probability  0  FMs  activate 
on  one  more  test  after 
stopping 


Probability  0  FMs  activate 
on  one  more  test  after 
stopping 
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Mean  Number  of  Tests  To  Reach  Requirement 


Mean  Initial  Number  FMs  Each  Stage:  6 
Probability  FM  Activate  by  Stage:  Q, 1,0, 1,0, 9, 0,9 
Pooled  Stage  Approximation 
Mean  Initial  NumberofFMs=Sum;Average  Activation 
Probability 
200  Replications 


♦4  Stages:  Consecutive 
Tests  SR 

1 4  Stages:  Non- 
consecutive  tests  SR 

A  Pooled-Stage: 
Consecutive  Tests  SR 

*  Pooled-Stage:  Non- 
consecutive  Tests  SR 


NumberTests  Until  No  FMs  Activate 
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Probability  No  FMs  Activate  On  Next  Test 

4-Stage  System: 

Mean  Initial  Number  FMs  Each  Stage:  6 
Probability  FMActivationbyStageiO.  1,0. 1,0, 9,0, 9 
PooledStage  Approximation 
Mean  Number  FMs=24;  Average  Activation  Probability 
200  Replications 


1  2  3  consecutive  Tests  SR 

NumberTests  Until  No  FMs  Activate 
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Probability  0  FMs  Activate  on  Next  Test 
Stop  After  R  "Successes" 

4-Stage  System: 

Meant  FMs  Each  Stage:  0,2, 0.2,6, 6 
Probability  FM  Activate  by  Stage:  0,1, 0,1,0, 9,0, 9 
Pooled  Stage  Approximation 
Mean  Number  FMs=Sum;  Average  Activation  Probability 
200  Repl. 


NumberTests  Until  No  FMs  Activate 


Mean  Number  of  Tests  To  Reach  Requirement 
4-Stage  System: 

Mean  Initial  Number  FMs  Each  Stage:  0,2, 0,2, 6,6 
Probability  FM  Activate  by  Stage:  0,1, 0,1, 0,9, 0,9 

PooledStage  Approximation 

Mean  Number  of  FMs=Sum;  Average  Activation  Probability 
200  Repl. 


4  4  Stages:  Consecutive 
Tests  SR 

1 4  Stages:  Non- 
consecutive  tests  SR 

A  Pooled-Stage: 
Consecutive  Tests  SR 

t  Pooled-Stage:  Non- 
consecutive  Tests  SR 


NumberTests  Until  No  FMs  Activate 
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Fixed  Number  of  Tests 

Mean  #  FMs  Each  Stage:  0.2, 0.2, 6, 6 
Probability  FM  Activation  by  Stage:  0.1,0. 1,0. 9, 0.9 

Pooled  Stage  Approximation:  Mean  #  FMs=Sum; 

Average  Activation  Probability 

200  Replications 


♦  Pooled  Stage;  Average 
Activation  Probability 


♦  Multiple  Stages; 
Masking 


Summary 


•  The  1-Stage  (Pooled)  System  can  be  optimistic  compared  to 
system  with  MASKING 

-  Smaller  Mean  Number  of  Tests  Until  Obtain  the  Required 
Number  of  Successes 

-  Larger  Probability,  next  test  activates  no  FMs,  each  stopping  rule 

•  R  Consecutive  Successful  tests  versus  R  Successful  tests 

-  Larger  number  of  tests, 

BUT 

-  Larger  probability  one  more  test  will  not  activate  FM 

•  Fixed  Number  of  Tests  may  not  be  enough 

-  Testing  to  Learn 
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Reference 


•  D.  P.  Gaver,  P.  A.  Jacobs,  K.  D.  Glazebrook,  and 
E.  A.  Seglie.  "Probability  models  for 
sequential-stage  system  reliability  growth  via 
failure  mode  removal"  International  Journal 
of  Reliability,  Quality  and  Safety  Engineering, 
10  (2003),  15-40. 


19 


SOA  Testing  Tools 


Army  Testing  in  a  Services  Oriented  Architecture  (SOA)  Environment 


Army  Proven 

Bottle  Ready 


Michael  Phillips 
254-287-8258 
ManTech  International 
Michael.Scott.Phillips@us.army.mil 


Instrumentation  of  the  Past 


Data 

Collector 


4 


Real-time  Feed 

or  (RICS)2  Log  File 


Database 


Message 


Traffic  tactical  Switch 
Span  Port 


DPU 


Data  Reduction 
and  Analysis 
(DRA) 


Tactical  System 


Tactical  System 


•  Testing  Army  computer  systems  before  SOA 

-  Collection 

•  Attach  to  LAN  and  collect  everything  •  Promiscuous  non-intrusive  methods 

-  Reduction 

•  Revolved  around  the  parsing  of  formatted  message  traffic 

-  Protocols  -  Message  standards 

-  Analysis 

•  Metrics  were  essentially  constant 

-  Speed  of  Service  -  Message  Completion  Rate 

Army  Proven 

Battle  Ready 


-  Message  Standards  Compliance 

US  Army  Electronic  Proving  Ground 
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Evolution  of  Instrumentation 


•  In  the  2000s,  changes  in  the  Army  Battle  Command  Systems 
drove  changes  in  instrumentation  methodologies 

-  Joint  Common  Database  (JCDB) 

■  First  attempt  to  maintain  a  common  database  by  conducting  database  replication 
between  information  systems  within  a  TOC 

■  EPG  developed  new  data  collection  methodologies 


-  Data  Collection  Module  (DCM)  developed  as  an  Embedded  Agent 


-  Army  Information  Server  (AIS) 

■  First  Publish  and  Subscribe  Service  (PASS)  architecture  for  intra-TOC  exchanges 

-  Fixed  topic  assignments  for  pub/sub  (1 6  topics) 

-  No  advertising  -  subscribers  had  to  poll  to  discover  new  topics 

-  ABCS  provided  stove  pipe  comms  for  interoperability  between  TOCs 

■  EPG  developed  new  Stimulation,  Data  Collection,  and  Visualization  tools 

-  Bulk  PASS  as  a  Surrogate  Client  to  publish  and  subscribe  to  the  server 

-  PASS  Data  Collector  (PDC)  as  a  Surrogate  Client  to  capture  encrypted  exchanges 

-  PASS  Monitor  as  a  Custom  Visualization  Tool  for  validation  of  transactions 


Army  Proven 


Battle  Ready 


US  Army  Electronic  Proving  Ground 


Current  Testing  Environment 


Data  Dissemination  Service  (DDS) 

■  Replaces  AIS 

■  Introduces  topic  advertising  (64  DDS  advertising  profiles) 

■  Queries  and  dynamic  subscriptions 

■  Introduces  Server-to-Server  Peering 

■  With  DDS  all  LAN  traffic  is  encrypted 

Instrumentation  Requirements 

■  Validate  DDS  server  operation 

■  Validate  client  publications  against  standards 

■  Monitor  JCR-DDS  Interactions 
EPG  Developed  Solutions 

■  Modify  existing  Surrogate  Clients 

•  Utilize  DDS  Client  Interface  (DCI) 

•  Incorporate  SDK  from  PM  Battle  Command 

■  Modify  existing  Embedded  Agent 

■  Modify  existing  Custom  Visualization  Tool 

■  Developed  a  method  to  Decrypt  Network  Data 

■  Incorporate  Logs  from  the  System  Under  Test  (SUT) 


Soon,  SOA  will  replace 
the  majority  of 
message  exchanges 


DDS  was  the  beginning  of  a  move  to  Services  Oriented  Architecture 


Army  Proven 

Battle  Ready 


US  Army  Electronic  Proving  Ground 


Intro  to  SO  A 


Service  Oriented  Architecture 


Army  Proven 

Battle  Ready 


US  Army  Electronic  Proving  Ground 


Impact  of  SOA 


•  SOA  features  will  change  current  test  paradigms 

■  Encryption 

•  Most  LAN  traffic  will  be  encrypted 

•  Listening  promiscuously  is  no  longer  feasible 

■  Thin  Clients 

•  Standalone  applications  gone,  replaced  by  services 

•  Most  message-based  communications  obsolete 


Army  Proven 

Battle  Ready 


US  Army  Electronic  Proving  Ground 


6 


Intro  to  Virtualization 


The  intent  of  using  virtual  systems  is  to 
utilize  increases  in  computer  horsepower 
to  reduce  the  number  of  physical 
systems  necessary  in  an  architecture. 

It  also  allows  systems  to  be  easily 
interchanged  while  avoiding  installation 
problems. 


Hardware  (CPU,  Memory,  NIC,  Disk) 

Hypervisor  (Hyper-V,  Xen,  ESX  Server) 


lflB| 

Application 


g  ■  jk  l  y  a  ■  s 

Application  Application 


Guest  OS 


Guest  OS 


Virtual  Hardware 


Virtual  Hardware 


Guest  OS 


Virtual  Hardware 


' 


All 


Army^Proven  _  US  Army  Electronic  Proving  Ground 

Bottle  Ready 


Intro  to  Virtualization  Cont. 


Application  Software  - 

OS  Installed  on  Virtual  Hardware  — 

Virtual  Hardware  - 

Virtual  Machine  - ^ 

Hypervisor  software  (VMWare,  etc.) 

Real  Server  Hardware  - 


Hardware  (CPU,  Memory,  NIC,  Disk) 

Hypervisor  (Hype r-V,  Xen,  ESX  Server) 


pplication 


Application 


Application 


est  OS  Guest  OS 


Guest  OS 


Army  Proven  US  Army  Electronic  Proving  Ground 

Battle  Ready 
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Impact  of  Virtualization 


Hardware  (CPU,  Memory,  NIC,  Disk) 

Hypervisor  (Hype r-V,  Xen,  ESX  Server) 


* 

Application 

Application 

W 

Application 

d  *  * , 

Guest  OS 

1 

Guest  OS 

Guest  OS 

Virtual  Hardware 

Virtual  Hardware 

Virtual  Haidware 

L  *- 


In  an  environment  with  virtualized  systems,  this 
may  be  just  a  monitor,  keyboard,  and  mouse,  or 
it  may  be  another  computer.  Either  way,  there 
are  no  data  on  the  wires  between  it  and  the 
server  hardware. 


Data  transmitted  between  the  server  stack  and  other 
systems  in  the  local  or  remote  network  will  traverse 
standard  network  equipment  and  be  available  for 
passive  LAN  collection  at  the  switches. 

Army  Proven 

Battle  Ready 


US  Army  Electronic  Proving  Ground 
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Impact  of  Virtualization 


Hardware  (CPU,  Memory,  NIC,  Disk) 

Hypervisor  (Hype r-V,  Xen,  ESX  Server) 


Data  transferred  between  virtual  systems 
hosted  on  the  same  server  stack,  however, 
never  leaves  the  virtual  environment  and 
cannot  be  captured  by  a  hardware-based 
collector. 


Army  Proven 

10  Battle  Ready 
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The  Future  Army  Architecture 

Company  Command  Post  Transport  Layer 


NetMgmt 

S1PR 


BC  “Collapse”  Strategy 


NCW 

HNW 


COMMS 

•  Interchangeable  transport  based 
on: 

•  Organic  assets  at  Company 

•  ESB  available  assets 
brought  to  Company 


ROUTING 

•  Abstracts  the  transport 

•  Take  advantage  of  WIN-T 
routing 


a 

< 


■Jr^- 

■sPH- 

H 


-sfH  -*=ri 

I  BCS3  L-  -'I _ J 


|  Bi  6 
|  Ei»i 


BC  Services 

E  nvii  oiimern 

BCSwvic  |H  1  [1 

Fi.vlio.im«a 

^ - 

- 

Existing  instrumentation  will  not  meet  the  Army’s  needs 
These  architectures  will  begin  testing  at  the  CTSF  very  soon 

August  2011 
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BC  “Collapse”  Strategy 
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In  Five  Years,  no  more  standalone  applications  in  the  TOC 


BCTM  Company  Command  Post 


NCW 

WNW 

HNW 

NCW 

WNW 

HNW 


SINCGARS 

SRW 


EPLRS 
SRW  / 

ANW2 
SINCGARS 

SINCGARS 

SRW 

SRW 

Other 

NCW 

HNW 


NetMgmt 

SIPR 


VM  Server  Stack 


VOIP 

Phone 


MS  Office 


Work 

Station 

(NIPR) 


NetMgmt  NIPR 


Information  Systems  pushed  down  to  the  CO  CP  level. 

■  Virtual  systems  within  a  single  VM  Server  Stack. 

Black  lines  carry  NO  data. 

■  Grey  boxes  in  the  picture  represent  only  monitors  and  keyboards. 
Intra-TOC  comms.  will  be  invisible  to  hardware-based  data  collectors. 


ATEC  ToolKit 


Testing  these  systems  will  require  a  multi-tool  approach 


Embedded  Agents 


Virtualized  Data  Collecto 


Custom  Visualization  Tools 


Surrogate  Clients 


z 


Decrypt  Network  Data 


System  Under  Test  (SUT)  Logs 


S  SOA  Testing  Tools 
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Collection  &  Reduction  Simulation 


EPG's  SOA  Tools 


DDS  Server 


Captures 

system 

metrics 


Bulk  PASS  acts  as  a  BFA  client  to: 
Advertise 
Publish 
Retract 
Subscribe 
Locally  or  Globally 


Analysis  Tools  read  DPU 
db  then  display  and 
validate  DDS  exchanges 


BFA  Client 


DPU  decrypts  LDC  data  LAN  Data  Collector  (LDC)  PASS  Data  Collector  (PDC) 
DPU  parses  LDC  and  PDC  data  collects  encrypted  DDS  acts  as  a  BFA  client  to 

and  creates  SQL  db  data  exchanges  and  Subscribe  to  all  DDS  events  and 

sends  to  DPU  sends  to  DPU 


Army  Proven 

Battle  Ready 


US  Army  Electronic  Proving  Ground 


Bottom  Line 


Current  Instrumentation 

-  Collection 

•  Attach  to  LAN  and  collect 
everything 

•  Promiscuous  non-intrusive 
methods 

-  Reduction 

•  Revolved  around  converting  raw 
data  into  something  useable 

-  Protocols 

-  Message  standards 

-  Analysis 

•  Metrics  were  essentially  constant 

-  Speed  of  Service 

-  Message  Completion  Rate 

-  Standards  Compliance 


Army  Proven 

Battle  Ready 


SOA-Compatible  Instrumentation 

-  Collection 

•  LAN  data  important  but  not  primary 

-  Requires  decryption 

•  Active  data  collection  methods 

-  Surrogate  Clients  and  Embedded  Agents 

-  Requires  Cooperation  with  PMs 

-  Early  involvement  in  process 

•  Flexibility  Required 

-  New  methodologies 

-  Custom  solutions  for  each  test 

-  Reduction 

•  Revolves  around  the  big  picture 

-  Conformance 

-  Data  flow 

-  Integration 

-  Analysis 

•  New  Metrics  will  be  developed 

-  Yet  to  be  determined 

-  Likely  to  change  rapidly 

US  Army  Electronic  Proving  Ground 
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TENA  and  JMETC  Enabling  Interoperability 
Among  Ranges,  Facilities,  and  Simulations 


Briefing  for: 

NDIA  27th  Annual  National  T&E  Conference 


March  16,  2011 

Gene  Hudgins,  TENA  SDA  User  Support  Lead 


What  is  JMETC? 


*  A  corporate  approach  for  linking  distributed  facilities 

•  Enables  customers  to  efficiently  evaluate  their  warfighting  capabilities 
in  a  Joint  context 

•  Provides  compatibility  between  test  and  training 

*  A  core,  reusable,  and  easily  reconfigurable  infrastructure 

•  Consists  of  the  following  products: 

•  Persistent  connectivity 

•  Middleware 

•  Standard  interface  definitions  and  software  algorithms 

•  Distributed  test  support  tools 

•  Data  management  solutions 

•  Reuse  repository 

*  Provides  customer  support  team  for  JMETC  products 
and  distributed  testing 


JMETC  Enables 
Distributed  Testing 


Joint  Operational  Scenarios 


Systems 

Under 

Test 

Integrated 

Test 

Resources 


Virtual 

Prototype 


TENA 

Standard 

Interface 

Definitions 


TENA 

Common 

Middleware 


Hardware 
in  the 
Loop 
Lab 


TENA 

Standard 

Interface 

Definitions 


TENA 

Common 

Middleware 


Installed 

Systems 

Test 

Facility 


TENA 

Standard 

Interface 

Definitions 


TENA 

Common 

Middleware 


Range 


TENA 

Standard 

Interface 

Definitions 


TENA 

Common 

Middleware 


Environment 

Generator 


TENA 

Standard 

Interface 

Definitions 


TENA 

Common 

Middleware 


Threat 

Systems 


TENA 

Standard 

Interface 

Definitions 


TENA 

Common 

Middleware 


Customer  Support 


JMETC:  Here  and  Now 


*  Uses  the  Secure  Defense  Research  &  Engineering 
Network  (SDREN)  for  connectivity 

•  61  sites  currently  on-line 

< _ X 

*  Uses  Test  &  Training  Enabling  Architecture  (TENA) 

•  Gateways  to  link  to  existing  DIS  and  HLA  simulations 

*  Incorporates  InterTEC  test  tools 

*  Uses  the  JNTC-sponsored  Network  Aggregator  to  link 
together  other  networks 


*  Being  expanded  based  on  customer  requirements 

*  Holding  JMETC  Users  Group  meetings  to  discuss 
emerging  requirements  and  technical  solutions 

•  Seeking  the  “best  of  breed”  solutions  across  the  community 
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oeing:  Tukwila 
eyport:  NUWC 


JMETC  Connectivity 

O  Functional  Sites:  61 
New  Sites  Planned:  6 
Connection  Points  to  Other  Networks:  4 


£ 

K 


/Ft  Lewis:  EPG 


Sites  in  SoCal 

O Edwards:  Ridley 

O  China  Lake  (3): 
AV-8B,  F/A-18, 
I  BAR 

OPoint  Mugu  (2): 

A  ITEC,  AEA 
Xj  El  Segundo: 

NGC  B-2 

)  Camp  Pendleton: 

MCTSSA 
C^jona:  NSWC 

)  Point  Loma  (2): 
RLBTS 
SSC-PAC 
Rancho  Bernardo, 
NGC  BAMS 
West  Agg  Rtr. 


A  Sites  in  Hawaii 
A^PMRF:  Bldg  105 
O  MHPCC 


Dedicated,  trusted  connectivity  on  SDREN  (part  of  the  GIG) 

Encrypted  for  Secret  -  System  High 

DISA-registered  IP  address  space 

Active  monitoring  of  network  performance 

Capable  of  supporting  multiple  simultaneous  test  events 

T 


Tinker  AFB:  AW  ACS 


Redstone  (6):  RTC:  DTCpfGMAN 
SED:  Patr4etrTHAAD,  FAAD,  SIV 


Ft.  Sill:  FSED 


-C 


WSMR:  IRCC 

*_J 


Ft  Huachuca:  (2)  JITC,  JTDL 


Site  in  Alaska 

Ft.  Greely:  CRTC 


Greenville:  Rivet  Joint 

o 


CTSF,  TTEC 


J  M  ETC:  site  E:parnlon 


□  Sites  : 

_ J _ 1=1 

Sites  in  Gulf  Range 
OHurlburt  Field:  C2DAC 
OEglin  AFB  (4): 

AOC,  DTF,  GWEF, 
KHILS 


Melbourne: 
NGC  JSTARS 

\J 


Hanscom  AFB:  CEIF 

u-iUipage:  NGC  BAMS 

Ft.  Monmouth:  JOIN 

Sites  in  MD,  DC,  VA 
)A^rdeen:  ACCN 
Pax  River  (5): 

O  ESTEL  E2C/D,  MCL 
_  ACETEF,  SAIL,  ATR 
u  JMETC  SYSCON 
East  Agg  Rtr. 

^  O  Pentagon:  WARCAP 
O  DISA:  Sky  7 
ODahlgren  (2):  CEDLJWSL 
OJFCOM:  JSIC 
O  Langley  (2): 

C-GIIF,  TDLITC 

§  Norfolk:  COMOPTEVFOR 
Norfolk:C2F  Mitscher  Ctr 
QDam  Neck:  CDSA 
OWallops  Island: 

(2)  SCSC,  SSDS 
Newport  News: 

NGC  VASCIC 
McLean:  MITRE  NCEL 


As  of  09  Dec  2010 


®  Army 
O  Air  Force 
O  Navy 
•  Marines 
O  Joint 
O  Industry 


*1 


JMETC:  Here  and  Now 


•  Uses  the  Secure  Defense  Research  &  Engineering 
Network  (SDREN)  for  connectivity 

•  61  sites  currently  on-line 

\ 

•  Uses  Test  &  Training  Enabling  Architecture  (TEN A) 

•  Gateways  to  link  to  existing  DIS  and  HLA  simulations 

_ _  _ / 

•  Incorporates  InterTEC  test  tools 


*  Uses  the  JNTC-sponsored  Network  Aggregator  to  link 
together  other  networks 

*  Being  expanded  based  on  customer  requirements 

*  Holding  JMETC  Users  Group  meetings  to  discuss 
emerging  requirements  and  technical  solutions 

•  Seeking  the  “best  of  breed”  solutions  across  the  community 


6 


JMETC  Uses  TENA  to  Integrate  Sites 

(Can  gateway  to  existing  DIS  and  HLA  simulations) 


•  TENA  is: 

•  Developed,  upgraded,  and  sustained  by  CTEIP  and  JNTC 

•  Middleware  that  provides  a  single,  universal  data  exchange  solution 

•  Common  for  test  and  for  training  (core  standard  in  JMETC  and  JNTC) 

•  Available  for  download  at  www.tena-sda.org  for  free 

•  TENA  provides: 

•  Interoperability  among  range  systems,  hardware-in-the-loop  laboratories, 
and  simulations  in  a  quick,  cost-efficient  manner 

•  A  capability  to  rapidly  and  reliably  develop  LVC  integrations 

•  A  set  of  community-agreed  object  models  that  define  the  data  elements 
used  in  LVC  integrations  -  maximizes  reuse  from  event  to  event 

•  An  auto-code  generator  to  drastically  reduce  TENA  incorporation  time 

•  Newest  version  of  TENA  (version  6.0)  provides: 

•  Advanced  data  filtering  (only  data  of  interest  sent  over  the  wire) 

•  Improved  fault  tolerance  and  embedded  diagnostics 

•  Downloadable  on  the  TENA  Website 


h 


JMETC 


Gateway  Builder 


•  GWB  is  focused  on  integration  of  distributed  live,  virtual,  and  constructive 
(LVC)  systems  into  a  common  synthetic  battle  space  that  comprises  various 
simulation  protocols,  training  ranges,  live  systems  and  platforms 

•  Gateway  Builder  streamlines  integration  process  and  reduces  time  and 
effort  of  creating  gateways 


MSR  Gateway  Builder 


Select  Protocols 


Map  Protocol  OMs 


INe  I 

-  Bemen 

-  Bemen 

-  Hr  inrn 


|  Paige  B 

^ClnCJil  1 


DC 

-  Berne 


•  H  TEN 

•  Berne  nl  1 

-  Berne  nl2 

-  Berne nl3 

-  Berne  nU 

-  Bemenl* 

-  Bemenl* 


ML* 

-  Be  me  nl  l 

-  Bemenl2 

-  Bemenl  3 

-  Be  menu 

-  Bemenl* 

-  Bemenl* 


IM  el 

TEN* 

-  Bemenl  1  _ 

-  Bemenl  1 

-  Bemenl  2 

Bemen  12 

-  Bemenl  3—, 

Bemen  13 

-  Bemenl  4*-1 

--  Bemen  U 

-  Bemenl  5 

- 

-  Bemen  15 

-  Bemenl* 

-  Bemenl* 

HL* 

-  Bemenl  I 

-  Bemenl2 
menl3. 


iemeni3--  >  - 

le  tre  nU  ^  * 

Be  me  nl*  yZ!~.-  J  * 
Be  me  nl*''T 


TEN* 

,*  Bemenl  l 
_*  Bemenl  2 
-  Bemenl  3 
Bemenl  4 
Bemenl  * 
Bemenl  G 


Generate 

Gateways 


Gateway  Builder  is  a  flexible,  extensible,  graphically  driven  tool  that 
automatically 
generates  gateways 
to  bridge  simulation 
and  live  protocols 

Gateway  Builder 
supports  mappings 
between  TENA,  DIS, 
and  HLA  and 
message-based 
protocols  using 
any  object  model 


Gateway  Builder  Simplified  Block  Diagram 


12  Oct  2006 
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fENA  Overview 


Requirements 

-  Interoperability 

-  Reuse 

-  Composability 

-  Support  Rapid  Integration 

-  Gradual  Deployment 


Guiding  Principles 

-  Provide  middleware 

-  Use  real  software  objects 

-  Maximize  code  generation 

-  Management  by  users  (AMT) 

-  No  license  fee  (GOTS) 


•  Supports 


-  Testers  &  Trainers 

-  Joint,  Army,  Navy,  Air  Force,  Agencies 

-  Live,  Virtual,  Constructive 

-  Range,  Laboratories,  Simulations 

-  Real-Time  &  Non-Real-Time 


TENA  Applications  IZIZZZZ:  TEN  A  tools' 


TENA  Utilities 


Non-TENA  Applications^ 


TENA  Architecture  Overview 


TENA  Applications 


TENA  Utilities 


Non-TENA  Applications 
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Key  Release  6  Improvements 
and  New  Capabilities 


New  Middleware 

Metamodel  and  Model 

Capabilities 

Improvements 

■  Advanced  Filtering 

■  Fundamental  Sized  Type  Aliases 

■  OM  Subsetting  Support 

■  Const  Qualifier 

■  SDO  State  Processing  Support 

■  Optional  Attributes 

■  Self-Reflection  Option 

■  SDO  Initializers 

■  Object  Reactivation 

■  Middleware  Metadata 

■  Separate  Inbound/Outbound  ORBs 

■  Middleware  IDs 

New  Event  Management 

Usability 

Capabilities 

Improvements 

■  Object  Model  Consistency  Checking 

■  Observer  Pattern 

■  Remote  Object  Termination 

(with  Callback  Aggregation) 

■  Execution  Manager  Fault  Tolerance 

■  Local  Methods  Factory 

■  Embedded  Diagnostics 

Registration 

■  TENA  Console 

■  Code  Installation  Layout 

11 


Key  Release  6  Improvements 
and  New  Capabilities 


New  Middleware 

Metamodel  and  Model 

Capabilities^-^^7  \ 

lmprovement§^-^^  \ 

■  Advanced  \ 

■  V^ctrSte  Inbound/Outbound  ORBs 

■  Fundamental^^^fin©  \ 

■  Con>^VtO  det'  J 

■V^JcTreware  IDs 

New  Event  Managemeirt^~\ 
Capabilitj£i^^^  \ 

Usability 

Improvemep^^^  \ 

'  ObjecUVIodP^iVitV  fing  J 

r^pTO'/  ifO0^^ie^ce 

\.en^Sr 

■  Observer^p^^  \ 

^  •H^^^rrTation  Layout 
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TENA  in  a  Resource  Constrained  Environment 

(TRCE)  S&T  Background 


•  Low  Data  Rate  Networks 

-  TENA  must  be  able  to  establish  and 
maintain  data  connections  on  low  data 
rate  networks 

-  Need  to  optimize  use  of  low  data  rate 
networks  to  support  relevant 
operational  scenarios 

•  Wireless  Networks 

-  Current  range  environments  use 
wireless  links  extensively  for  various 
systems  under  test 


•  Variable  Quality  Networks 

-  T&E  systems  poorly  tolerate  high  loss,  link 
failure,  or  heterogeneous  links 

-  Need  to  provide  data  continuity  for 
degraded  or  heterogeneous  networks 

•  Specification  of  Interests 

-  Subscribers  must  be  able  to  specify  data 
“interests”  to  more  efficiently  use  available 
&  limited  network  resources 


TRCE  Phase  1  will: 

•Developed  Use  Cases  and  Requirements 

•Developed  Proof-of-Concept  Applications  to  Investigate  Candidate  Technologies 
•Quantified  Benefits  of  Candidate  Technologies 
•Representative  Laboratory  Environment 
•Successful  Phase  1  Technology  Demonstration 

•Recommended  Technologies  for  Further  Development  and  Inclusion  in  the  TENA 
Middleware 


TRCE  is  providing  TENA  for  variable  quality  and  low  data  rate  network 
links  including  wireless  networks 


13 


Wireless"Networ 
(Ufoan  Environm  i 


JMETC  VPN 


Hands-on  TRCE  Current  “Smartphone” 
Capabilities  at  the  TENA/JMETC  Booth 


•  Booth  Demonstration  Capabilities  Using  TENA  RelayNode  and  TENA 
Video  Distribution  System  (TVDS)  with  iPads  and  iPod  Touch  Devices 

-  Display  of  Platform  positions  on  static  maps  stored  locally  on  the  handheld  devices 

-  Selection  and  real-time  viewing  of  available  video  streams  managed  by  TVDS  on 
handheld  devices  (iPhone/iPad/Android) 

-  Pan/Tilt  control  of  remote  cameras  (and  firing  of  Nerf  remote  “missile  launcher” )  via 
TENA  remote  methods 


•  Highlights  the  Flexibility  of  TENA  Middleware 


-  Remote  control  of  instrumentation  via  TENA  Remote  Methods 

-  Use  of  wireless  networks  including  3G 

-  Middleware  implementations  on  small  form  factor  computers  such  as  Smartphones 


TENA  and  RRRP 


>  Use  of  TENA  will  facilitate  Remote  Operations  and  Interoperability  of  the 
Ranges’  Radar  Systems 

>  TENA  Instrumentation  Radar  Object  Models  will  be  used  for  all 
communications  external  to  the  individual  Radar  Systems 

-  Pointing  data  for  optics,  telemetry,  or  other  radars 

-  Remote  Single  Integrated  Air  Picture  (SIAP) 


>  Development  of  TENA  Instrumentation  Radar  Object  Models 


-  Developed  initial  Instrumentation  Radar  TSPI  Object  Model 

-  Received  input  from  Test  Center  SMEs 

-  For  CW  Doppler  and  Pulse  radar  systems 

-  Instrumentation  Radar  Object  Models  will  be  finalized  after  contract  award 


nsrjptwn 


20x  TSPJj  Accuracy 


I  m  provement  Level  1 1 


Miniaturization 


Technology 


>20x  TSPI 
Accuracy 
Improvement 
Level  III 


Data  Iftroughputelx 
Improvement,  So 


mcsiijorj 


Standardized  Protocols 


Improved  Reliability 


3x  TSPI  Accuracy  Improvement 
Level  I 


Systems 


Common  Range  Integrated  Instrumentation  System 


^Trajflfng  (RIW) 
^Vaveform  with 

S Training  Level 

,  , . 


Ground  Station/  stat/on 


Subsystem  (TENA) 


Live  Monitorina 


JSTEIPj 


Alaska  Training  Range  Evolution 
Program  (ATREP)  use  of  TENA 


ATREP’s  intent  is  to  enhance  the  existing  Pacific  Alaska  Range 
Complex  air  and  ground  capabilities  by  providing  a  force-on- 
force  (FOF)  training  capability  that  fully  integrates  and  supports 
oint  and  coalition  components  for  both  air  and  ground  training 
n  live,  virtual,  and  constructive  (LVC)  domains. 


High  Side 
•TENA  ICADS 
•TENAACMI 
•TENA  9C2 
•TENA  DIADS 
•TENA  SimShield 

Low  Side 
•TENA  MOKKITS 
•TENA  MILES  2000 
•TENA  l-HITS 
•TENA  UMTE 


c 
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Partial  Listing  of  Recent  Testing, 
Training,  and  Experiments 
Using  TENA-Compliant  Capabilities 


Test  Events 

•  SIAP  JDEP  Combined  Hardware-in-the-Loop  Phase  5, 
Jan-May  09 

•  Digital  Close  Air  Support  -  Integrated  Model  Test 
Event,  Jan-Mar  09 

•  Multi-Service  System-of-Systems  Test-bed,  Jul  09 

•  Strategic  Integrated  M&S  Capability,  May-Aug  09 

•  Joint  Electronic  Warfare  Assessment  for  Test  and 
Evaluation,  Sep  09 

•  Tactical  End-to-End  Closed  Loop  Sim,  Nov  09 

•  Joint  Distributed  IRCM  System  Test  Event,  Mar  10 


•  Training  Exercises 

•  Daily  Training,  Eielson  AFB 

•  Daily  Training,  Fallon  AFB 

•  Red  Flag  Alaska  (RFA)  09-1 ,  October  08,  Pacific 
Alaska  Range  (PARC) 

•  JDEWR  Cope  Tiger  09,  Mar  09,  PARC 

•  RFA  09-2,  April-May  09,  PARC 

•  Distant  Frontier,  May-June  09,  PARC 

•  Northern  Edge  09,  June  09,  PARC 

•  Talisman  Sabre  09  -  Australian  Army  and  US 
Army,  July  09,  Shoalwater  Bay,  Queensland 
Australia 


•  Joint  Close  Air  Support  Distributed  Test,  Jun  10 

•  Battlefield  Airborne  Communications  Node  (BACN) 
Joint  Urgent  Operational  Need  (JUON),  Aug  10 

•  JIAMDO  Air  &  Missile  Defense  Correlation  / 
Decorrelation  Interoperability  Test  (CDIT)  CONUS, 
Sept  10 

•  Unmanned  Aircraft  System  (UAS)  in  National  Air 
Space  (NAS)  Oct  09  and  Oct  10 

•  JITC  Joint  Interoperability  Test  (JIT)  Sep-Nov  10 

•  JIAMDO  CDIT  UK,  Oct  10 

•  Air-to-Ground  Integrated  Layer  Exploration  AGILE  Fire 
III,  Feb  11 


•  RFA  09-3,  July-Aug  09,  PARC 

•  JDEWR  Talisman  Sabre  09,  July  09,  PARC 

•  RFA  1 0-1 ,  October  09 

•  RFA  10-2,  April  10 

•  Northern  Edge,  June  10 

•  RFA  10-3,  Aug  10 

•  Experiments 

•  Joint  Surface  Warfare  JCTD,  Feb  09  and  Oct  10 

•  Joint  Expeditionary  Force  Experiment  (JEFX) 

09-1, 09-2,  09-3,  Feb-Apr  09 

•  JEFX  09-4  B-2  Test  (Spirit  ICE),  Aug  09 

•  JEFX  10-1,  10-2,  10-3,  Jan-Apr  10 


Distributed  Events  operated  over  the  JMETC  and  JTEN  Connectivity 


*1 


JMETC:  Here  and  Now 


*  Uses  the  Secure  Defense  Research  &  Engineering 
Network  (SDREN)  for  connectivity 

•  61  sites  currently  on-line 


*  Uses  Test  &  Training  Enabling  Architecture  (TEN A) 

•  Gateways  to  link  to  existing  DIS  and  HLA  simulations 

/ - \ 

*  Incorporates  InterTEC  test  tools 

\ _ / 

*  Uses  the  JNTC-sponsored  Network  Aggregator  to  link 
together  other  networks 

*  Being  expanded  based  on  customer  requirements 

*  Holding  JMETC  Users  Group  meetings  to  discuss 
emerging  requirements  and  technical  solutions 

•  Seeking  the  “best  of  breed”  solutions  across  the  community 
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InterTEC  Operational  View-1 

TENA-Based  Integrated  Test  Tool  Applications 


JMETC 


C4ISR  Instrumentation  20  lnte9rated  APPS  in  SPiral  2 


Test  Control 


&  Analysis  Joint  C4ISR  Test 


•Planning 


Rehearsal 

Control 

Monitoring 

Reporting 


Environment 


Data  Capture 
Stimulation 
Analysis 
Display 


Virtual  Components 
•HWIL 
Interfaces 
*  •  Message 

Generation 


Live  Components 

•  Range 
Interfaces 
•Range  * 
Instrumentation 


Constructive 

Components 

•  Simulation 
Interfaces 


InterTEC  Integration  with  JMETC 
Inextricably  Intertwined 


4Q  2010 

InterTEC 
Spiral  3 

4Q  2008 

InterTEC 
Spiral  2 

Fielded - 

InterTEC 
Spiral  1 


IBS,  UAV, 
CGS  &  SOA 
Test  Tools 


OTH-G  & 
USMTF 
Test  Tools 


Link-16, 
Link-11, 
&VMF 
Test  Tools 


JMETC 

Toolbox 


JMETC 

Toolbox 


JMETC 

Toolbox 


JMETC  supports  InterTEC  during  their  spiral  development 
InterTEC  expands  JMETC  toolbox  with  certified  C4ISR  Test  Tools 


TENA  Integrated  Development 
Environment  (TIDE) 

•  TIDE  is  a  tool  designed  to  assist  developers  in  the 
creation,  development,  testing  and  deployment  of  TENA 
applications 

•  Initial  Capabilities 

-  Catalog  installed  object  models  on  a  user’s  machine 

-  Migrate  user  applications  between  object  model  versions 

-  Migrate  user  applications  between  middleware  versions 

-  Browse  and  download  object  models  available  in  the  TENA  Repository 

-  Request  object  model  distributions  from  the  TENA  Repository 

•  TIDE  2.0  is  the  current  version 

-  Available  at  http://www.tena-sda.org/tide  web  site 
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TENA  Tools  used  by  JMETC 
Interface  Verification  Tool  (IVT) 


*  Designed  to  support  the  integration  testing  of  TENA 
applications 

-  TENA  Standard  OM’s 

-  JNTC  and  InterTEC  LROM’s 

*  Provides  real-time  monitoring,  logging  and  statistics 
gathering 

*  Operates  in  three  different  roles,  either  stand-alone  or  in 
combination: 

-  Data  Subscriber  Role 

-  Data  Publisher  Role 

-  DIS  to  TENA  Gateway  Role 

*  Available  at  https://www.tena-sda.org/displav/Tools/IVT 
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SIMDIS  UseofTENA 


SCORE  TSPI  Feed 


.£  3  35 


£  ./SCORE  15PIS1MAICRSHWET  I 


_ Active  Streams  _ 

ao^t  _ Star! 

2S3S  "  10  Mar  03  07  33  08 


i»  U  /  £ 


Southern 

California 


NRL 

Washington,  DC 


TENA 


Duration  testing  using  SCORE  TSPI  data  feed 

•  Four  consecutive  days 

•  Win  XP,  Red  Hat  9,  Solaris  5.8 

•  Processed  180,000+  entities 

•  Two  consecutive  days 

•  Win  XP,  Red  Hat  9 

•  Processed  53,000+  entities 

Results  and  observations 


SIMDIS 

1 . -  / 

-JL. 

. 

Seeker  to  Warship 

Gain:  6.28(dB),  Pwr:  -141.S4(dBm) 

No 

No 

No 

No 


ssues  with  discovery  latency 
ssues  with  update  latency 
ssues  with  CPU  usage 
ssues  with  memory  usage 


SIMDIS 


ir 
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JMETC:  Here  and  Now 


*  Uses  the  Secure  Defense  Research  &  Engineering 
Network  (SDREN)  for  connectivity 

•  61  sites  currently  on-line 


*  Uses  Test  &  Training  Enabling  Architecture  (TENA) 

•  Gateways  to  link  to  existing  DIS  and  HLA  simulations 


*  Incorporates  InterTEC  test  tools 
- \ 

*  Uses  the  JNTC-sponsored  Network  Aggregator  to  link 

together  other  networks _ 

*  Being  expanded  based  on  customer  requirements 


*  Holding  JMETC  Users  Group  meetings  to  discuss 
emerging  requirements  and  technical  solutions 

•  Seeking  the  “best  of  breed”  solutions  across  the  community 
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Traditional  Network  Dilemma 


29  Palms 

Luke  AFB 

*1 


Network  Aggregation 
Bridging  Networks 


Network  Aggregation 
Bridging  Networks 


if] 
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Available  Sites  from  Integration  of 
Test  and  Training  Networks 
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JMETC:  Here  and  Now 


*  Uses  the  Secure  Defense  Research  &  Engineering 
Network  (SDREN)  for  connectivity 

•  61  sites  currently  on-line 


*  Uses  Test  &  Training  Enabling  Architecture  (TENA) 

•  Gateways  to  link  to  existing  DIS  and  HLA  simulations 


*  Incorporates  InterTEC  test  tools 


*  Uses  the  JNTC-sponsored  Network  Aggregator  to  link 
together  other  networks 

*  Being  expanded  based  on  customer  requirements 

*  Holding  JMETC  Users  Group  meetings  to  discuss 
emerging  requirements  and  technical  solutions 

^  •  Seeking  the  “best  of  breed”  solutions  across  the  community  ^ 


31 


*1 


JMETC  Users  Group  Meetings 


•  Identify  core  infrastructure  requirements 
and  use  cases 

•  Identify,  investigate,  &  resolve  issues 

•  Identify  opportunities  to  collaborate 

•  Discuss  available  solutions, 
tools,  and  techniques 

•  Share  lessons  learned 


Users.  Group  #0.1 


Users  Group  #02 

Users  Group  #03 

I  Isars  f^rnun  #0^ 


Users  Group  #05 


Users  Group  #06 

Users  Group  #07 

Users  Group  #08 

Next  JMETC  Users  Group 

Meeting  #13: 

•  Scheduled  for  22-23  March 

•  Location:  Norfolk,  VA 

•  Potential  Tracks: 

•  User  Requirements 

•  Information  Assurance  /  Security 

•  Data  Management 

•  Networking 


Users  Group  #09 

Users  Group  #10 

•23-24  Feb  2010 

•  Orlando,  FL 

•  ~300  participants 

•  Plenary  session: 

•  TRMC 

•  NavyT&E 

•  Tracks: 

•  User  Requirements 

•  IA  /  Security 

•  Object  Models 

•  Networking 

•  SOA 


Architecture  Management  Team 

(TENA  AMT) 


AMT  Members: 

•  329  Armament  Systems  Group  (329  ARSG) 

•  Aberdeen  Test  Center  (ATC),  Aberdeen  Proving  Ground,  MD 

•  Air  Armament  Center  (AAC),  Eglin  AFB,  FL 

•  Air  Force  Flight  Test  Center  (AFFTC),  Edwards  AFB,  CA 

•  Army  Operational  Test  Command  (OTC),  Fort  Hood,  TX 

•  Common  Training  Instrumentation  Architecture  (CTIA) 

•  Dugway  Proving  Ground  (DPG) 

•  Electronic  Proving  Ground  (EPG) 

•  integrated  Network  Enhanced  Telemetry  (iNET) 

•  Interoperability  Test  and  Evaluation  Capability  (InterTEC) 

•  Joint  Fires  Integration  &  Interoperability  Team  (JFIIT) 

•  Joint  National  Training  Capability  (JNTC) 

•  Naval  Air  Warfare  Center  -  Aircraft  Division 

•  NAWC  -  Weapons  Division 

•  Naval  Aviation  Training  Systems  Program  Office  (PMA-205) 

•  Naval  Undersea  Warfare  Center  (NUWC) 

•  NAVSEA  Warfare  Center  -  Keyport 

•  P5  Combat  Training  System  (P5CTS) 

•  Pacific  Missile  Range  Facility  (PMRF) 

•  Redstone  Technical  Test  Center  (RTTC) 

•  T&E/S&T  Non-lntrusive  Instrumentation 

•  White  Sands  Missile  Range  (WSMR) 


r. 


Meetings  every 
3  months 


US  Advising  Members: 


•  BMH  Associates,  Inc. 

•  Boeing 

•  Cubic  Defense 

•  DRS 

•  Embedded  Planet 

•  EMC 


Kenetics 

MAK  Technologies 
NetAcquire 

Science  Applications  International  Corporation 
(SAIC) 

Scientific  Research  Corporation  (SRC) 
Scientific  Solutions,  Inc.  (SSI) 


International  Participation 

•  Australia 

•  Denmark 

•  France 

•  Singapore 

•  Sweden 

•  United  Kingdom 


Design  Decisions  /  Trade-offs  /  Status  /  Technical  Exchanges  of  Lessons  Learned  /  Use 
Cases  /  Testing  /  Issues  &  Concerns  Identification,  Investigation  &  Resolution 


Summary 


•  JMETC  supports  the  full  spectrum  of  Joint  testing,  supporting  many 
customers  in  many  different  Joint  mission  threads 

•  CVN-21,  JSF,  MM  A,  NECC,  DD1000,  WWF,  BAMS,  JIAMDO 

•  TENA  is  the  CTEIP  architecture  for  future  instrumentation,  the  JNTC 
architecture  for  Live  integration  and  an  enabling  technology  for 
JMETC 

•  TENA  and  JMETC: 

•  Being  built  based  on  customer  requirements 

•  Partnering  with  Service  activities  and  leveraging  existing 
capabilities 

•  Coordinating  with  JFCOM  to  bridge  test  and  training  capabilities 

•  Provide  a  forum  for  users  to  develop  and  expand  the 
architecture 

•  JMETC  User  Groups,  TENA  AMT  Meetings 

•  Next  Meeting  is  week  of  March  21  in  Norfolk,  VA 
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Important  Contact  Information 


•  TENA  Website:  www.tena-sda.org 

•  Download  TENA  Middleware 

•  JMETC  Website:  www.imetc.org 

•  TENA  Feedback:  feedback@tena-sda.org 

•  Provide  technical  feedback  on  TENA  Architecture  or  Middleware 

•  JMETC  Feedback:  imetc-feedback@imetc.org 

•  TENA  SDA  Contact 

•  Telephone:  (703)  601-5202 

•  JMETC  Program  Office  Contact 

•  Telephone:  (703)  601-5280 
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Tri-Service  Study  2011 

27th  Annual  National  Test  and  Evaluation 
Conference 
Tampa,  FL 
15  March  2011 


Minh  Vuong 
DETEC  Project  Director 
(407)  384-5238  (DSN:  970) 
i  Minh.Vuong@us.army.mil 


Doug  Weatherford 
T-SS  Lead,  PEO  STRI 
(407)  384-5258 

Doug.Weatherford@us.army.mil 


Distribution  Statement  A:  Approved  for  public  release;  distribution  is  unlimited. 


DETEC  is  funded  by  the  Central  Test 
and  Evaluation  Investment  Program 


DETEC  -  Directed  Energy  Test  and  Evaluation  Capability  MRTFB  -  Major  Range  and  Test  Facility  Base 
(DEW  -  Directed  Energy  Weapon  TRMC  -  Test  Resource  Management  Center  T&E  -  Test  and  Evaluation 


Develop  Joint  T&E  MRTFB 
infrastructure  required  for 
T&E  of  DEW  systems 

-  Instrumentation 

-  Equipment 

-  Software  tools 

DEW  systems  supported 

-  High  energy  laser  (HEL) 

-  High  power  microwave  (HPM) 

Coordinate  T&E  needs  with 
TRMC  S&T  efforts 


DETEC  Mission 
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DETEC  History 


T-SS  Begins 

to  identify  and 
scope  shortfalls  in 
DET&E 
infrastructure 


T-SS  Update  (2007)  Begins  to 

capture  current  DE  T&E 
infrastructure  shortfalls  for 
DETEC  II,  originally  planned  to 
start  in  2010 


T-SS  2011  Begins 

to  capture  most  current  DE  T&E 
infrastructure  shortfalls 


2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014 


DETEC  Effort 
Proposed  to  CTEIfr 


DETEC  Begins  to  resolve  12  high  priority  capabilities 
identified  by  the  T-SS.  DETEC  is  developing  16  systems 
(covering  40  shortfalls)  to  address  these  infrastructure 
shortfalls 


Projected  Window  to  Begin  to 

resolve  the  highest  priority 
shortfalls  identified  by  the  T-SS 
2011 


DET  S&T  Begins  to  address  high-risk,  high-pay-off  shortfalls 
identified  by  the  T-SS.  To  date,  DET  S&T  has  delivered  20 
systems 
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T-SS  Overview 


Tri-Service  Study 

-  Objective:  identify  DE  T&E  infrastructure  shortfalls 
emphasizing  current  changes  to  baseline 

-  Goal:  reduce  DE  weapons  programs'  need  to  pay  for  T&E 
infrastructure;  prevent  delays  to  programs  awaiting  T&E 

Scope 

-  T&E  infrastructure  unique  to  DE  testing 

•  HEL  and  HPM  domains 

•  Impacts  all  test  activities  -  modeling  and  simulation  (M&S), 
developmental  T&E  (DT&E),  operational  T&E  (OT&E),  and  live-fire 
T&E  (LFT&E) 

•  Across  all  phases  of  a  test  event  (planning,  rehearsal  and 
execution,  analysis) 

•  Blue  DEW  vs.  Red  target  and  Red  DEW  vs.  Blue  target 

-  Leverage  existing  MRTFB  infrastructure 
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T-SS  Process 


'fa  Denotes  Modeling  and  Simulation  Working  Group  (MSWG)  Participation 
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Capability  Call 


Capability  Call  consists  of  four  questionnaires  to  the  DE  community  to  assess 
what  DE  T&E  capabilities  currently  exist 

Returned  questionnaires  were  entered  into  the  database  for  comparison  with 
DE  T&E  requirements  identified  through  Service  Workshops 

Over  30  completed  questionnaires  received  from  20+  organizations 
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Solution  Call 


Solution  Call  requested  a  short  white  paper  from  the  DE  community 
Government,  industry,  and  academia  participated 

Responses  help  DETEC  determine  cost,  schedule,  and  risk  of  T-SS  2011  identified 
shortfalls 
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SAWG  and  SRG  Review 


Services  met  on  31  October  to  validate  shortfalls  Identified  by  the  T-SS  2011 

Senior  Analyst  Working  Group  (SAWG)  met  on  19  November  to  establish  initial 
priority  of  shortfalls 

Senior  Review  Group  (SRG)  composed  of  an  SES  from  each  service  and  a 
representative  from  the  Electronic  Combat  (EC)  Reliance  Panel  met  on  20  January 
to  finalize  the  T-SS  2011  shortfall  priority 
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T-SS  2011  Results 


# 

Domain 

Capability  Shortfall 

1 

HPM 

Non-intrusive  E-field  and  B-field  probes 

2 

HPM 

X-band  surrogate  narrowband  threat  source 

3 

HEL 

CW  irradiance  measurement  on  surface  moving  target  board,  towed  airborne 
target  board,  and  actual  target 

4 

HPM 

C-band  surrogate  narrowband  threat  source 

5 

HPM 

Multiple  node  wireless  data  acquisition  system 

6 

HEL 

Imagery  of  airborne  targets 

7 

HEL 

Front  target  surface  temperature 

8 

HEL 

Dynamic  hazard  analysis  tool  (M&S) 

9 

HEL 

Predictive  avoidance  and  airspace  deconfliction  tools  (M&S) 

10 

HPM 

Beam  propagation  in  and  near  surfaces  (M&S) 

11 

HPM 

THP/Builder  integration  (M&S) 

Conclusion 


T-SS  2011  identified  11  high  priority  capability 
shortfalls 

In  process  of  documenting  final  results  and 
delivering  to  the  Test  Resource  Management 
Center  (TRMC) 

Collecting  Service  endorsements  of  the  T-SS 
2011  results 
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^Cambridge 

^^Consultants 


Holographic  radar  brings  a  new 
dimension  to  sensing  and 
instrumentation  on  T&E  ranges 

Collision  avoidance,  wind  farms  and  scoring 


NDIAtest  and  evaluation  conference 

Gary  Kemp 


2  March  2011 


S4923-P-069  v0.2 


Cambridge 

Consultants 


Short  introduction  to  Cambridge  Consultants 
What  is  holographic  radar? 

Applications  of  holographic  radar 
Questions 


<#> 


2  March  2011 


S4923-P-069  v0.2 


Overview  of  Cambridge  Consultants  -  Corporate  Overview 


A  world  leader  in  technology  and  product  development 


■  Established  in  1960 

■  300  engineers  and  scientists  based  in 
Cambridge  UK  and  Cambridge  MA. 

■  We  serve  a  wide  range  of  industries  -  defence, 
wireless,  transport,  consumer,  industrial, 
medtech 

■  We  design,  develop  and  manufacture 
innovative  products,  processes  and  systems 
using  multi-skilled  teams 

■  We  have  a  long  track  record  of  technology 
based  spin-out  companies 

■  We  manufacture  and  supply  specialist  radar 
systems 
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Cambridge 

Consultants 


Short  introduction  to  Cambridge  Consultants 
What  is  holographic  radar? 

Applications  of  holographic  radar 
Questions 


<#> 
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What  is  Holographic  radar? 


^Cambridge 

^^Consultants 


Holographic  radar  implements  Skolnik’s  vision  of  Ubiquitous  Radar 


■  Holographic  Radar  looks  continuously  at  a  whole  volume  of  space  (rather  than 
scanning). 

■  It  acquires  fully  sampled  amplitude  and  phase  information  from  every  object  within 
the  volume. 

■  It  provides  range,  azimuth,  elevation  and  Doppler  information  for  every  detected 
object. 

■  Tracking  algorithms  discriminate  moving  targets  and  clutter. 

■  Clutter  is  removed  without  loss  of  sensitivity. 


■  Practical  holographic  radar  is  possible  in  the  modern  day  due  to  the  availability  of 
high-power  processor  devices  at  reasonable  cost. 
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Cambridge 

Consultants 


Short  introduction  to  Cambridge  Consultants 
What  is  holographic  radar? 

Applications  of  holographic  radar 
Questions 
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^Cambridge 

^^Consultants 


Collision  warning  radar 


5-channel  array  for  automotive  pre-crash  sensing  -  a  minimum  holographic  array 
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Collision  warning  radar 


□ 


Cambridge 

Consultants 


2  March  2011 
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Wind  farms  and  Primary  Surveillance  Radar 


Many  wind  farm  planning  applications  are  stalled 


Aircraft 


Wind  farm 


Absence  of  vertical 
discrimination  combined  with 
scan  aliasing  makes  it 
impossible  for  a  PSR  to  separate 
the  track  from  the  clutter. 

Holographic  radar  provides  the 
solution. 
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Wind  farm  infill  radar 
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CH-InFill  is  a  holographic  radar  located  at  or  near  a  wind  farm  to  generate  local, 
high-resolution,  3D  infill  data 


■  The  sensor  is  located  in  or  near 
the  wind  farm 

■  It  sees  through  and  around  the 
turbines  without  disruption 

■  Nothing  else  has  been  shown  to 
do  this 


■  Range  up  to  13km  /  43,000ft 

■  Reporting  rate  3-1 0Hz 
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Wind  farm  infill  radar  -  testing 


66m  diameter  wind  turbine 


Remote-controlled 
helicopter  with  2.2m2  radar 
reflector 
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LSTS 


Land  and  Surface  Target  Scorer  (LSTS)  system  -  in  development 


■  The  Land  and  Surface  Target  Scorer  is  a  real-time  vector  scoring 
system  for  highly  mobile  targets  operating  in  very  cluttered 
environments. 

■  LSTS  application  of  the  CH  radar  is  funded  by  the  OSD  Target 
Management  Initiative  program,  sponsored  and  managed  by  NAWC- 
WD,  Point  Mugu,  Target  Systems  Division,  5.3.1 


LSTS  sensor  head  HSMST 


burst  .  -*  '  trajectory 


1000ft  scoring  volume 
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Range  (m) 


LSTS 


Two  views  of  how  LSTS  will  perform: 


Migration  process  rejects  clutter 
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Accuracy  and  throughput 


Range  (5”  Shell) 

50ft  -  1 000ft 

Firing  rate 

Up  to  20  rounds  / 
minute 

Along-track  position 
accuracy 

1 3ft  /  5%  at  longer 
range 

Target  speed 

Up  to  46kts  (at  SS3) 

Up  to  1 0Omph  (land) 

Sea  state 

Up  to  sea  state  3 

Trajectory  reporting 

Within  3  seconds  of 
projectile  arrival 
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LSTS 


Performance  measured  to  date: 


Cricket 

ball 

RCS 

0.004m2 


Radar 

head 


TV 


Cricket  bowling  machine  -  1 0Omph 
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LSTS 


Performance  measured  to  date: 


j**T.*t*«e*ri  C*&*t**wv*v*wm9  IX  *»wlr  -r 


SNR  at  80m  35dB  (25W  Tx) 

Proof  of  concept 
trajectory  processing 
takes  15  minutes 
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Detection,  tracking  and 
best-fit  3D  trajectory 

Launch 
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LSTS  program 

Proof  of  Concept  Phase  (system  design  and  single  face  build) 

-r 

■  Start  date: 

Jan  2010 

■T 

■  System  Requirements  Review: 

April  2010 

/ 

■  Preliminary  Design  Review: 

June  2010 

-T 

■  Critical  Design  Review: 

September  2010 

■  Test  Readiness  review: 

February  2011 

■  Proof  of  Concept  System  trials  with  5”  shell: 

March  201 1  (in  progress) 

Beta-prototype  phase  (complete  system  build  and  test) 

■  Start  date: 

April  2011 

■  Trials  with  50  cal  rounds: 

June  2011 

■  Sea  trials  on  HSMST: 

December  201 1 
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Conclusions 


^Cambridge 

^^Consultants 


Holographic  radar  is  the  best  you  can  do  in  very  cluttered  environments 

"  Target  and  clutter  separation 

-  Continuously  gather  signals  from  a  large  volume  of  space 

-  Fully  sampled  amplitude  and  phase  data  from  every  target 

-  Separate  targets  of  interest  from  clutter  through  tracking  processes 

■  Applications  in  collision  avoidance,  PSR  infill,  scoring,  through-wall,  asset 
protection,  border  monitoring,  other... 

■  LSTS  system  under  development 

-  5”  and  50  cal  projectiles 

-  Land  and  sea  surface  targets 

-  Proof  of  Concept  sea  trials  underway 
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Short  introduction  to  Cambridge  Consultants 
What  is  holographic  radar? 

Applications  of  holographic  radar 
Questions 
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•  Increasing  System  /  Software  Complexity 

•  Increasing  T&E  Costs  or  more  Delivered  Defects 

•  What  can  T&E  do  to  be  more  Affordable 
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STHIN.TH  TH101  f«H  RMIVtl  A  TFi'YNJlOtaY 

Increasing  System  /  Software  Complexity 


•  The  environment  of  systems  is  ever  an  increasing 

complexity  with  more  and  more  software 

-  One  engine  manufacturer  predicts  that  the  cars  and 
trucks  of  2020  will  have  over  a  billion  lines  of  code 

•  It  appears  that  every  new  weapon  system  has 
more  and  more  software 

•  Testing  this  much  software  is  impossible  with 
current  methods 
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STHIN.TH  TH101  f«H  RMIVtl  A  PYHNliO'iY 

Increasing  System  /  Software  Complexity 


Some  Background 

Last  years  NDIA  T&E  Conference  presentation 
titled  “Closing  the  T&E  Gap...” 

-  “The  funding  and  research  fortesting  &  evaluating  new 
technologies  is  not  keeping  pace  with  the  rate  of 
technology  change” 


4 


27th  Annual  National  T&E  Conference 


March  2011 


The  Exponential  Times  We  Live  In 


STKIMmi  THB04  VM  MK*m  A  TTnMMDGY 


Current  Rates 


■  New  technology 
information  doubles 
every  2  years. 

■  Adoption  of  technology 
is  accelerating 

■  To  Reach  50  million  users 

■  Radio  38  years 

■  TV  13  years 

■  Internet  4  years 

■  IPOD  3  years 

■  Face  Book  2  years 


This  data  is  from  the  “Did  You 


Predictions 


■  By  2013,  a  super 
computer  exceeds 
capability  of  human 
brain 

■  By  2049  $1000  computer 
exceeds  the  capability 
of  the  entire  human 
species. 


'now”  series 
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Compute  Power  and  Bandwidth  Increasing 


Bio 

Nano 


1990 


2010 


2010 


Computing  Power 


Internet  Data 


1995  2002 


Bandwidth 


- 


The 

Information 

Revolution 


1990  1995  2002 


Internet  Usage 


1990  1995  2002 


1990  1995  2002 


Compute  capacity  continues  to  grow 

Hardware  limitations  no  longer  constrains  the  software 
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Where  is  Connectedness  Headed? 
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Artificial  Intelligence 

Personal 

Assistants 


Intelligent  Agents 


Semantic  Web 
Connects  Knowledge 


Semantic 

Webs 


Ontologies 


Taxonomies 


Knowledge 

Bases 


Knowledge 

Management 


Search  Engines 
Content  Portals 


Enterprise 

Portals 


Groupware 


Web  sites 

The  Web. '' 

Connects  Information 

PIM’s 

Databases  “Push” 

Pub-Sub 


File  Servers  ___ 

y^,''  P2P  File-sharing 


Knowledge 

Networks 


The  Global 
Brain 


Enterprise 

Minds 


Group 

Minds 


Lifelogs 


The  Metaweb 

Connects  Intelligence 

“The 

Relationship 
Web” 


Smart 

Marketplaces 


Semantic 

Weblogs 


Decentralized 

Communities 


Marketplaces 

Auctions 


Wikis 


Weblogs  RSS 

Social  Software 
Connects  People 


Community 

Portals 


E-mail 


Conferencing 


USENET 


IM 


Social 

Networks 


Permission  to  re-use  with  attribution  to:  Nova  Splvackwww.mlndingtheplanet.net 


Degree  of  social  connectivity 
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Diminishing  Constraints 


STUIXmi  TMOt  r.H  MtffIVY  A  TTlWHXOtA 


I 


t 


Self  Adapting, 
Collaborative  Systems 


Diminishing 
hardware  constrains 


Increasing 
software  complexity 

Systems 


Applications 


Anywhere  —Anytime 


O 


Calculations 
Small  Pr 


)bil  to  Hybrid 
evices 

2000  -2010 


Computers  to 
Internet 

1970  -  1990 


MO 

Relays  To 
Vacuum  Tubes 
1930  -  1950 


Transistors  to 
Microprocessors 

1950  -  1970 


Merging  of 
Biology, 

Nanotechnology 
and  Artificial 
Intelligent 

The  future 


Old  paradigm  of  testing  no  longer  effective 
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Increasing  T&E  costs  or  more  delivered  Defects 


•  The  cost  of  T&E  for  these  very  complex  systems  is 
increasing  significantly  when  there  is  a  real  attempt 
to  detect  defects  before  delivery 

•  My  observation  is  that  the  following  is  occurring: 

-  Some  test  automation  is  attempted  with  marginal  success 

-  Test  times  are  not  generally  being  increased  so  less  test 
coverage  of  functionality  is  actually  occurring 

-  Results:  more  (complex)  defects  are  being  delivered 

•  Constant  pressure  to  reduce  T&E  cost  &  schedule 
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Test  Technology  &  Research  Funding  Gaps 


What  about  T&E  S&T  and 
Corporate  I  RAD  projects? 


What  about  the  Technology 
Readiness  Level  Process? 


Bottom  Line:  T&E  is  not  able  to  keep  up  with  Technology 
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A  Question 


STKIXmi  THBOt  r.H  HIKBTTl  a  v  i  wnii  cm 


•  Is  all  this  complexity  necessary? 

-  Absolutely,  it  makes  the  Warfighters  more  effective, 
efficient  and  safer 

•  So  what  can  we  do  as  Test  and  Evaluation 
engineers? 
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T&E  must  become  more  Effective  NSlk 

STKI>4mi  TfikfM  VM  WMSIWI  A  TTlUMHCKiY 

•  Need  more  T&E  Research  in  Government, 

Industry  and  Academia 

-  Ensure  Research  is  coordinated  whenever  possible 
(i.e.  between  Services,  initiated  /  coordinated  by 
Government,  etc) 

•  Review  the  Technical  Readiness  Level 
processes  to  see  if  possible  to  add  official  T&E 
deliverables  with  TRL’s  at  specific  levels, 
especially  in  the  range  of  TRL  4  &  5. 
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T&E  must  become  more  Effective 

STHINiTH  THMfM  f«H  A  PYHNliO'iY 

•  Much  more  automation  should  be  in  use  today 

-  Training  and  investment  must  be  done  and  this  will 
have  a  big  return  on  investment 

•  Ensure  the  use  of  Scientific  (or  Analytical)  Test 
Design  methods  and  tools 

•  Ensure  T&E  is  actively  involved  with  the 
research  and  related  activities  in  the  Model- 
Based  Development/Engineering  arena 
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Q  &  A 
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Overview 


*  Testable  Requirements 

*  Why  Reliability  Growth? 

*  Reliability  Growth  Models 

*  PM2  -  Assumptions 

*  PM2  -  Parameters 

*  PM2  -  Under  the  Hood 

*  F-15E  Radar  Modernization  Program  (RMP)  Example 

*  Conclusions 
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Testable  Requirements 


*  Testable  requirements  must  be 

-  Realistic 

-  Measurable 

-  Possible  to  evaluate 

*  Sometimes  requirements  are  stated  as  a  future  need 

-  We  can’t  evaluate  to  a  point  in  the  future 

-  The  decision  authority  must  decide  now  if  the  system 
should  be  acquired 

*  AFI  10-601 

-  “If  the  production  threshold  value  is  planned  to  be 
achieved  following  completion  of  IOT&E,  include  a  testable 
value  to  be  achieved/demonstrated  for  evaluation  during 
the  IOT&E.” 


\  k  i  ! 
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Why  Reliability  Growth? 


*  Reliability  is  the  ability  of  a  system  to  perform  required 
functions  for  a  specified  period  of  time 

-  Ex:  mean  time  between  critical  failures  (MTBCF)  for  RMP 

*  Reliability  growth  is  an  increase  in  system  reliability  as 
a  result  of  corrective  actions 

*  Time  between  IOT&E  and  when  the  system  must 
demonstrate  reliability  allows  for  growth 

*  Reliability  growth  plans  improve: 

-  Investment  decisions 

-  Operations  and  maintenance  posturing 

-  Assessment  of  progress  over  time 
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Reliability  Growth  Models 


*  Duane  Model  -  1964 

-  Logarithmic  growth 

-  Formalized  “test,  analyze,  and  fix”  process 

•  Crow-AMSAA  Model  -  1974 

-  Failures  as  a  stochastic  process 

-  Allows  for  statistical  evaluation  of  growth 

•  MIL-HDBK-189  - 1981 

-  DoD-specific  guidelines  for  planning 

-  Yardsticks  for  assessing  growth 

*  Planning  Model  based  on  Projection  Methodology  -  2006 

-  Introduces  parameters  based  on  programmatics 

-  Combines  programmatics  and  statistics 


5 


PM2  -  Assumptions 


*  The  number  of  failure  modes  present  in  the  system  at 
the  beginning  of  the  time  period  is  inherently  unknown 

*  Individual  failure  mode  occurrences  are  independent  of 
all  other  failure  mode  occurrences 

*  System  usage  and  environment  can  be  predicted 
throughout  the  reliability  growth  cycle 

*  Failures  follow  a  nonhomogenous  Poisson  process 

-  For  a  defined  period  of  time,  a  certain  number  of  failures 
are  expected 

-  Failures  can  happen  at  any  time 

-  Time  between  failures  is  independent 

-  As  reliability  improves,  the  failure  rate  decreases 


PM2  -  Parameters 


•  Define: 

-  What  is  the  required  end-state  of  the  system? 

■  Mg:  goal  reliability 

-  What  proportion  of  fixes  can  you  make? 

■  MS:  management  strategy 

-  How  effective  will  implemented  fixes  be? 

■  FEF:  fix  effectiveness  factor 

-  What  is  the  best  possible  state  the  system  can  achieve? 

■  K:  ratio  of  goal  reliability  to  growth  potential  reliability 

-  How  much  operating  time  will  the  system  accumulate? 

■  T:  total  time 
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PM2  -  Under  the  Hood 


A 

%  - 


M 


Determine: 

-  M,:  The  initial  reliability  that  enables  reaching  MG 


M,  >  (1  —  FEF  *  MS)  * 


M, 


K 


-  p:  The  planning  curve  shape  parameter 

M,  \ 


1 

*=T 


1  - 


M, 


MS  *  FEF 


-  M(t):  The  expected  reliability,  in  terms  of  cumulative  time 


M(t)  = 


M:(l  +  |3  *  t) 


1+  p  *  t  *  (1  -  MS  *  FEF) 
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F-15E  RMP  Example 


F-15E  RMP  Example 


\  ^  i  ! 

k  ,  i 


•  Key  performance  parameter:  radar  MTBCF  of  575 
operating  hours  at  full  operational  capability  (FOC) 

•  12-year  gap  between  IOT&E  and  FOC 

•  5  jets/1200  hours  for  IOT&E 

•  Parameters 

-  MG  =  575  operating  hours 

-  MS  =  0.9 

-  FEF  =  0.8 

-  K  =  0.8 

-  T  =  300K  operating  hours 

•  M|  ~  200  operating  hours 
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RMP  Growth  Curve 


a 


F-15E  RMP  -  PM2  Reliability  Growth  Planning  Curve 


2013-15 

2016-2018 

2019-2020 

2021-22 

2023-2024  (FOC) 

*  —Requirement 

Time  (Operating  Hours) 


11 


Conclusions 


*  Creating  a  reliability  growth  plan  allows  improved: 

-  User  planning  for  manpower  and  sustainment 

-  Program  office  programming  and  budgeting  activities 

-  Test  team  evaluation  of  realistic  and  measurable  metrics 

-  System  performance  assessment  in  a  transparent  and 
objective  manner 

*  The  earlier  the  planning  process  takes  place,  the  better 

*  Using  rigorous  statistical  methods  provides: 

-  Credible  and  defensible  results 

-  Powerful  techniques  to  assess  progress 

-  Quick  “what-if”  analysis 
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Questions 


m 
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Abstract 


*ni^rri  TtfinnairaKSTii  attihviu>.i 


In  2009,  the  NDIA  System  of  Systems  Committee  developed  a 
white  paper  describing  test  and  evaluation  issues  that  cause 

"sleepless  nights". 

In  2010,  the  NDIA  SoS  and  DT&E  Committees  collaborated  in  a 
joint  workshop  to  translate  these  issues  into  strategic  initiatives  and 
collaborative  go-do  activities  as  improvement  areas.  The  issues 
included  future  T&E  for  systems  brought  together  as  SoS, 
requirements,  metrics,  systems  changes,  and  end  to  end  testing 

with  systems  not  yet  available. 

This  paper  will  summarize  the  results  of  that  workshop  and  the 
progress  being  made  to  mitigate  SoS  T&E  sleepless  nights. 
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NDIA  DT&E  Committee 

Mif^m  mtiM  mNram  a  nsHVHj>.i 

Moved  from  SE  to  T&E  Division 


Focus  of  this  Paper:  DT&E  Collaboration  with  SoS 
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Sleepless  Nights: 

Test  and  Evaluation  for  SoS 


KTirarv  nttni  im*  vrn  a  TtiHMLntfr 


•  Systems  of  Systems  Topics  Discussed  in  2009: 

-  Compiled  list  of  “what  keeps  me  awake  at  night”  topics  for  SoS 

-  Test  and  evaluation  for  SoS  topped  the  “Sleepless  Nights”  list 


•  NDIA  SoS  and  DT&E  Committees  Worked  Jointly  in  2009: 

-  Identified  key  T&E  challenges  for  SoS 

-  White  paper  described  5  top  issues 

-  Presented  at  2009  NDIA  SE  Conference  in  joint  SoS/T&E  track 

•  Focus  for  2010:  Joint  Workshop  August  1 7th 

-  Define  a  path  from  Sleepless  Nights  to  Sominex 

-  Evaluate  challenges  and  underlying  issues 

-  Transition  specific  issues  into  strategic  initiatives 

•  Resulting  Effort: 

-  3  Strategic  Initiatives 

-  1  Collaborative  Go-Do 

Workshop  Defined  Path  to  Find  Sleep  Aids 
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Reminder  from  2009: 
T&E  Challenges  for  SoS 


KTirarv  nttni  tat  im*  vrn  a  TtiHMLntfr 


1)  Future  T&E:  If  SoS  are  not  programs  of  record  (and  not  subject  to 
T&E  regulations)  why  should  we  worry  about  this  at  all? 

2)  Requirements:  If  „requirements"are  not  clearly  specified  up  front 
for  a  SoS,  what  is  the  basis  for  T&E  of  an  SoS? 

3)  Metrics:  What  is  the  relationship  between  SoS  metrics  and  T&E 
objectives? 

4)  Systems  Changes:  Are  expected  cumulative  impacts  of  systems 
changes  on  SoS  performance  the  same  as  SoS  performance 
objectives? 

5)  End  to  End  Testing:  How  do  you  test  the  contribution  of  a  system 
to  the  end  to  end  SoS  performance  in  the  absence  of  other  SoS 
elements  critical  to  the  SoS  results?  What  if  systems  all 
implemented  to  their  specification,  but  the  overall  SoS  expected 
changes  cannot  be  verified? 


White  Paper  was  Starting  Point 
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Facilitated  Workshop: 
The  Technique 


nftiH  im*  vm  a  ti  i  MViLn».T 


Data  Collection: 

SoS  White  Paper 
SE  Conference  Papers 


Potential  Problem  Areas 

1)  Future  T&E  for  Systems 
brought  together  as  SoS 

2)  Requirements 

3)  Metrics 

4)  Systems  Changes 

5)  End  to  End  Testing  with 
systems  not  yet  available 

Potential  Causes 
If  we  could  only  fix  one  thing, 
it  would  be 


Improvement  Areas: 
Strategic  Initiatives 
Collaborative  Go-Do 


Leverage  Matrix 

Map  Causes  to  problem  areas 


Transition  from  Problem  Space  to  Solution  Space 


NDIA  T&E  Conference  Mar  201 1 


6 


Facilitated  Workshop 
Attendees 


nftiH  tat  im*  vm  a  Hi 


Mr.  Robert  Aaron 

Army 

Government 

Dr.  JoAnn  Lane 

USC  CSSE 

Industry 

Col  (Ret)  Suzanne  M.  Beers 

MITRE 

FFRDC 

Mr.  Steven  S.  Lee 

DoD 

Industry 

Dr.  William  D.  Bell 

MITRE 

FFRDC 

Mr.  Marty  Leek  (Facilitator) 

Raytheon 

Industry 

Mr.  Aumber  Bhatti 

MITRE 

FFRDC 

Mr.  Favio  L.  Lopez 

Army 

Industry 

Clyneice  Chaney 

MITRE 

FFRDC 

Mr.  John  R.  Palmer 

Boeing 

Industry 

Mr.  Peter  H.  Christensen 

MITRE 

FFRDC 

Mr.  George  Rebovich  Jr. 

MITRE 

FFRDC 

Mr.  David  W.  Coleman 

MITRE 

FFRDC 

Mr.  Frank  J.  Serna 

Draper 

Industry 

Dr.  Judith  S.  Dahmann 

MITRE 

FFRDC 

Mr.  Michael  Shanahan 

USMC 

Government 

Ms.  Indira  Deonandan 

MIT 

Government 

Dr.  Carol  A.  Sledge 

SEI 

FFRDC 

Mr.  John  W.  Diem 

OSD/  MSCO  Government 

CDR  (Ret)  James  D  Smith  II 

SEI 

FFRDC 

Mr.  Mark  E.  Fenicle 

DoD 

Government 

Mr.  Thomas  Wissink 

Lockheed  Martin 

Industry 

Mr.  Tanya  Gobel 

SAIC 

Industry 

Mr.  Jack  Zavin 

OSD  NII/DoD  CIO 

Government 

Mr.  Robert  Heilman 

DOD 

Government 

Dr.  Janice  A.  Ziarko 

MITRE 

Industry 

CDR  (Ret)  Bryan  Herdlick 

JHUAPL 

Government 

Ms.  Robin  E.  Ziradinovic 

SAIC 

Industry 

Government 

28% 

Industry 

36% 


FFRDC 

36% 
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Workshop  Results 


Kimcri  nftt  *1  at  imk  vrn  a  n  i  HViij*.T 


Initiative  Title 

Action  Plan 

Initiative  Vision  Statement 

Strategic  Initiatives 

Best  Practices 

Model  for  SoS  T&E 

Define  a  best  practices 
model 

SoS  T&E  as  a  continuous  improvement  process 
supporting  capabilities  and  limitations  information 
for  end  users  and  feedback  to  SoS  and  System 

SE  teams  toward  evolution  of  the  SoS 

Radical  Approach  to 
SoS  T&E 

Define  SoS  capability 
test  approach 

Rethink  T&E  of  systems  in  an  operational  context 
and  systems  interoperability  away  from  system 
testing  toward  integrated  capability  SoS  testing 

SoS  Governance 

Define  characteristics  of 
successful  SoS  T&E 

Indentify  the  process  by  which  we  can  change  and 
influence  the  governance  of  SoS.  Mature  and 
improve  templates  to  define  a  minimum  set  of 
characteristics  that  are  required  to  govern  SoS 

T&E  efforts 

Go-Do 

SoS  SE  Policy  and 
Guidance 

Recognize  and  employ 
SoS  guidance 

Ensure  that  guidance  or  SoS  SE  (DoD  SoS  SE 
Guide)  is  recognized  and  employed  on  growing 
number  of  SoS 

Initiatives  Identified  with  Action  Plans 
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Initiative  Teams 


Mll’HTTl  nttlH  Ui  IM* vm  A  Tf  i  HVltn».T 


#1 

Best  Practices 

Leads 

Judith  Dahmann,  (MITRE  &ASD  R&E/SE) 
Rob  Heilman  (TRMC) 

Team 

Members 

George  Rebovich,  (MITRE) 

Jim  Buscemi  (GBL&TRMC) 

Paola  Pringle  (Navy) 

Kent  Pickett  (MITRE) 

Chris  Scrapper  (MITRE) 

Aaron  Budgor,  (GBL  Systems,  TRMC) 

Laura  Feinerman,  (MITRE) 

Joe  Lucidi,  (Army  OTC) 

#3 

Governance 

Leads 

Bob  Aaron  (ATEC) 

James  Smith  (SEI) 

Team 

Members 

John  Palmer  (Boeing) 

Carol  Sledge,  PhD  (SEI) 

Robin  Zivadinovic  (JFCOM/Ctr) 

#2 _ 

Define  SoS  Capability  Test 


2  Initiatives  Launched,  Will  Feed  Results  into  3rd 
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#1 :  Best  Practices  Model 
Approach  and  Status 


nftt  m  ai  imk  vrn  a  n  i  HViLn».T 


1. 

2. 


3. 


4. 

5. 

6. 


Form  core  team  (Complete) 

Core  team  will  implement  activities 
Share  results  for  feedback  from  SoS  and  DT&E  committee 

Define  scope  (Complete) 

Focus  on  Acknowledged  SoS  (SoS  objectives,  management,  funding  and 
authority;  however  systems  retain  their  own  management,  funding  and  authority 
in  parallel  with  the  SoS) 

Investigating  potential  for  Directed  SoS  (SoS  objectives,  management,  funding 
and  authority;  systems  are  subordinated  to  SoS) 


Complete 
In  Process 
Planned 


Develop  a  draft  description  of  the  proposed  model 

Review  the  workshop  discussions  (Complete) 

Review  current  SoS  SE  guidance  on  T&E  (Complete) 

Framework  for  model  and  implementation  approaches  (In  Progress) 

Draft  model  description  and  circulate  for  review  (Planned) 


Review  use  cases  to  support  and/or  adapt  the  model 

Update  the  model  based  on  use  cases 

Review  and  assess  state  and  utility  of  the  model 


Identifying  T&E  inserts  into  SoS  Wave  Model 
Soliciting  Use  Case  Recommendations 
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#1 :  Best  Practices  Model 
Role  of  T&E  in  SoS  Models 


nftiH  t  at  im*  vm  a  ni  MViLn».T 


Trapeze  Model 


External  Environment 


Artifacts  that  apply  across  the  elements 

SoS  Planning  SoS  Master  Risks  and  Agreements 


Elements 


Mitigations 


Capability 

Objectives 

CONOPS 

Requirements 

Space 


Information 
About  Systems 
which  Impact  SoS 


Information 
about  Systems 
Updates 

Technical 
Baseline  Updates 


SoS  SE  Guide  Trapeze  Model 

-  “Assessing  Performance”  is  a  core 
element  of  SoS  SE 

SoS  SE  Artifacts 

-  Performance  Measures  and  Metrics 

Wave  Model 

-  SoS  T&E  begins  with  SoS  analysis  and 
is  addressed  throughout  the  other  steps 

Wave  Model 


Technical  Plans 

Integrated  Master 
Schedule 

Performance  Data 


External  Environment 


Conduct 
SoS  Analysis 


External  Environment 


Continue 
SoS  Analysis 


C  a nf  i  ni  i a 

continue 

SoS  Analysis 

Plan 

SoS 

Update 


Implement 

SoS 

Update 


Implement 

SoS 

Update 
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#1 :  Best  Practices  Model 
Framework  for  Description 


nftt  m  ai  imk  vrn  a  n  i  hviu*.t 


SoS  Wave  Model 


•  Describe  key  activities  at  each  stage 
as  they  relate  to  T&E  of  the  SoS 

-  Conduct  (and  Continue)  SoS  analysis 

-  Develop  and  evolve  SoS  architecture 

-  Plan  SoS  Updates 

-  Implement  SoS  Updated 


External  Environment 


•  What  actions  are  taken  at  each 
step  to  support  the  model  of  SoS 
T&E  as 

“Continuous  improvement 
process  supporting  capabilities 
and  limitations  information  for 
end  users  and  feedback  to  the 
SoS  and  system  SE  teams  toward 
evolution  of  the  SoS” 


Why  are  these  important? 

What  value  to  they  add? 

How  do  they  contribute  to  the 
larger  SoS  SE  and  T&E 
outcomes? 

How  do  they  address  the 
challenges? 


•  What  methods  or  tools  apply? 
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#3:  Governance 
Approach  and  Status 


MKIl 

hi^tii  nittM  lairMHYrn  a  niHViii>.T 


1.  Form  core  team  (Complete) 

2.  Define  scope  (Complete) 

-  Purpose:  to  provide  an  integrated  governance  perspective  for  SOS 
development,  deployment,  and  life  cycle 

-  Scope:  Governance  for  overall  acquisition,  including  T&E  as  a 
holistic/comprehensive  view  (focus  on  Directed  and  Acknowledged  SoS) 

3.  Identify  Governance  As-ls  State  (Complete) 

-  Fundamental  Governance  Concepts 

-  Architecture  Concepts  &  DODAF  for  managing  complexity 

4.  Develop  Governance  To-Be  Fundamental  Concepts  (In  Process) 

-  Organizations  that  produce  reference  models,  reference  architectures,  and  data 
engineering  components  including  T&E  considerations  for  measuring 
performance 

-  Synchronized  and  aligned  organizations  (structures),  policy,  tools,  technical 
approaches,  and  resources  that  support  the  selected  option. 

5.  Draft  Recommendations  to  Achieve  To-Be  State 


Complete 
In  Process 
Planned 


Reference  Architecture  As  Framework  to 
Discuss  Governance 
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#3:  Governance 
Current  and  Objective  State 


nitm  ut  imk  vrn  a  m  wvitn'.T 


Our  Architecture/Technology  organizations  should  be  designed  how? 


Test  Technology  ^est  Technology 

Current  State  Objective  State 


/ - \ 

Reference  Model 


v 


r 

Reference 

- \ 

v _ 

Architecture 

_ / 

r 

vJ 

V 

A/  ^ 

Solution 

Architectures 

14 


Creating  the  Environment  for  NR  KPP  -  OV- 

(NR  KPP  WG  -  Draft) 


User  Needs 


Focus  of  major  changes 


Technology  Opportunities  &  Resources 


A 


A  Program 
Initiation 


Materiel 
Solution 
Analysis 

Materiel 

y  Development 


<L 


Decision 


•  The  Materiel  Development  Decision  precedes 
entry  into  any  phase  of  the  acquisition  framework 

•  Entrance  criteria  met  before  entering  phases 

•  Evolutionary  Acquisition  or  Single  Step  to  Full 
Capability 


A 


IOC 


FOC 


Technology 

Engineering  and  | 

Production  & 

Operations  & 

Development 

Manufacturing  Development 

Deployment 

Support 

PDR  /\ 

'  / 

/\  Post  PDR  /^Post-CDR 

s  / Assessment  Assessmerf 

FRP 

<  >  Decision 

V  Review 

or  _r 


3  Dec  2008 


3  Dec  2008 


Use  M&S 
Environment 

to  develop/test 
Reference 
Architecture 


Use  M&S 
Environment  to 

develop/test 

Solution 

CM  Architecture 


-  Begins  w.  Reference  Models 
-Technology/Cost  trade  offs 

-  Model  Driven  Architecture  (MDA) 

-  Integrated  Development  Environment  (IDE)  all  under  Configuration  Mgmt 

-  Use  M&S  Environment  to  develop  architecture  and  to  the 

extent  possible  SOS  DOTLMPS  components 

-  Common/interoperable  tool  kits  for  this  acquisition  domain 


-  User  pulls  down  what  he  needs 

-  Each  Phase  Virtual  using  DOD  COEs 

-  Designers  push  new  patches 


15 


EXAMPLE  ONLY 


Test  Technology  “To-Be”  System  Evolution  (SV-8) 


DTC 

1 

1 

OTC 

AEC 

(x9) 

1 

(x8) 

(x13) 

C)  \  )  i 

•Initial  Ref  Architecture 
for  TTD  Review 
•Initial  Reference 
Repository  with: 

•First  AEC  data  models 
•Search,  Submit,  & 
Download  Functions 


•Initial  Ref  Architecture 
for  Community  Input 
•Functioning  Reference 
Repository  with: 

•Initial  CRM  containing 
the  following  data 
models: 

•AEC  T&E  Reference 
Model  &  other  available 
data  models 
•TENA  TSPI 
•Engineering  Units  DB 


•M&S,  Instruments 
•Threat  Representations 
•Facilities/Infrastructure 
AT  EC/Army 
Standard  Interfaces 


VI&S 

•Instruments, 

•Threat  Representations, 
•Facilities/lnfrastructi  is. 
International, 

Joint,  Army,  &  ATEC 
\5tandard  Interfaces 


Common  Reference  Model 
within  the  ATEC  Reference 
Repository 


Common  Reference 
Model  within  the  ATEC 
Reference  Repository 


Common 

Databases 


•Ref  Architecture 
•Community  involved  in 
evolution 
•Initial  Automation  Support 
•Operational  Reference 
Repository  with: 

•Initial  Visualization 
functionality  for  data  model 
comparison 
•Enhanced  CRM  that 
includes  Planning  metadata 
•Additional  &  Evolved  data 
models 


•Ref  Architecture 
•Community  Active  in  Evolution 
•Improved  Automation  Support 
•Operational  Reference 
Repository  with: 

•Improved  Visualization, 
Aggregation,  Disaggregation 
functionality  for  data  models 
•Evolved  CRM  that  includes 
Execution  metadata 
•Additional  &  Evolved  data 
models 

•Initial  TT  Investment  Decision 
Support  System 


•Ref  Architecture 
•Community  Driven 
•Full  Automation  Support 
•Reference  Repository 
•Sophisticated  Visualization, 
Aggregation,  Disaggregation 
functionalityfor  data  models 
•Evolved  CRM  that  includes 
metadata  for  all  T&E  phases 
•Additional  &  Evolved  data 
models 

•Improved  TT  Investment 
Decision  Support  System 


CY08 


FY09 


FY10 


FY11 


FY12 


I 

More  Automation  -  Less  Manual  Effort,  Greater  Accuracy,  &  Less  Time 
Increasing  Community  Involvement,  Modeling  and  Governance  — 
Greater  Interoperability  &  Less  Ambiguity 
Growing  Architectural  Cohesion  Enabling  Informed  Decision  Making 


TEC  TTD  Architecture 


#2:  Capability  Testing 
Approach  Planned 


TUilM  Ui  IMrt  YTH  A  Til  HMUKr 


1.  Assess  inputs  from  Strategic  Initiatives  #1  and  #3 

2.  Form  core  team 

3.  Define  scope 

4.  Define  SoS  T&E  As-ls  State 

-  Build  up  of  systems  testing  in  operational  context 

-  Build  up  of  systems  interoperability 

5.  Define  SoS  Capability  T&E  To-Be  State 

-  Define  gaps  in  implementation  as  integrated  capability  SoS 

-  Identify  barriers  responsible  for  these  gaps 

6.  Draft  Recommendations  to  Achieve  Capability  SoS  T&E 


Complete 
In  Process 
Planned 


Rethink  T&E  of  SoS  in  Operational  Context 
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Summary 


KTvrvrni TUttH  Uf  IMAMU  A  Tf  i  HVlLn'.T 


•  Successful  Workshop  with  SoS  and  T&E  Practitioners 

•  Framework  Established  for  Continuing  Collaboration 

•  Transition  Discussion  from  Challenges  to  Solutions 

•  Strategic  Initiatives  to  Develop  T&E  Solutions  for  SoS: 

1.  Define  a  best  practices  model 

2.  Define  SoS  capability  test 

3.  Define  characteristics  of  successful  SoS  T&E 

-  Recognize  and  employ  existing  guidance  for  SoS  (DoD  SoS  SE  Guide) 


Not  Too  Late  to  Join  a  Team! 
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KTYTYrni TUttH  Uf  IMAMU  A  Tf  i  HVlLn'.T 


BACKUP 

Details  on  T&E  Issue  Discussions 
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Issue  1 


If  SoS  are  not  programs  of  record  (and  not 
subject  to  T&E  regulations)  why  should  we 
worry  about  this  at  all? 

Discussion 


•  Restatement  of  issue: 

-  How  do  we  define,  articulate,  and 
enforce  the  relationship  between  the 
SoS  and  the  constituent  systems? 

-  How  does  T&E  support/help  this? 

•  Governance/Roles/Stakeholders 

-  Need  a  shepard  (architect?)  and  support 
from  users 

-  Need  to  educate  stakeholders 

-  What  are  rules  of  governance?  , 

-  What  are  the  regulations,  standards,  and 
policies? 

-  Need  to  obtain  resources  (funding,  test 
assets,  time) 

-  SoS  leadership  focus:  architecture 
views,  who  “owns” 

-  Potential  conflicts  between  SoS  and 
constituents 

-  Business  case  for  PMs  to  do  SoS 


SoS  T&E  Focus 

-  SoS  T&E  operationally  driven  (vs.  DT-ish) 

-  SoS  edge  of  the  envelop 

—  What  is  an  AoA  of  SoS? 

-  Emergent  behaviors  (good  and  bad) 

-  SoS  resource  consumption  (e.g.  data  pipeline) 

—  Continual  assessment  (joint  exercises, 

deployments) 

—  How  to  define  test  strategies  to  efficiently 
continuously  test? 

-  How  do  we  help  the  T&E  process  help  the  SoS 
work? 

Understand  SoS  Capabilities 

-  What  is  the  SoS  expected  to  do? 

-  Define  and  articulate  relation  between  SoS  and 
systems 

—  Flexible  composition 

-  Artfully  sub-optimize  the  systems  in  favor  of  the 
SoS 

-  System  performance  bounds  are  not  rigid  in  real 
operation 

•  Candidate  solution:  SoS  requirements  document 
with  annex  for  each  constituent  system  (what  is 
constituent  contribution  to  SoS  capability) 
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Issue  1  If  SoS  are  not  programs  of  record  (and  not 

subject  to  T&E  regulations)  why  should  we 
worry  about  this  at  all? 

Approach  to  addressing  issue 

•  Define  a  minimal  set  of  SoS  governance  characteristics  of  a 
successful  acknowledged  SoS 

-  Roles/resources 

-  Rules/regs/standards/policies 

—  Managing  conflicts 

—  Establishing  cooperation  of  constituent  systems 

-  Includes  responsibility  to  define  SoS  capabilities,  architecture,  and 
associated  test  strategy 

—  Concept  of  continual  change  and  test  in  operational  and  training 
environment 

—  Lean  management,  taking  advantage  of  available  opportunities 

—  Recognize  the  large  number  of  SoS  across  the  DoD,  and  the  fact  that 
many  systems  support  multiple  SoS.anf  the  potential  impacts  of 
governance 
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ISSU6  tt2  If  “requirements”  are  not  clearly  up  front 

from  a  SoS,  what  is  basis  for  T&E  of  an  SoS? 

Discussion 


Requirements  vs  expectations;  Mission 
objective  vs.  technical  requirements 

Mission  threads  linked  to  capability 
strands  as  architecture  model 

Who/what  has  responsibility  for 
architecture/requirement-  another  DOD 
layer? 

Standards  for  participating  or  acceptance 
of  each  system  into  SoS 

Requirements  model  for  architecture 
encompassing  time,  space  changes 

SoS  level  requirement  T&E  at  program  or 
SoS  level  balance? 

T&E  of  aggregation  of  systems  level 
requirements  (SOS  level  TEMP) 

Integrated  development  environment/ 
reference  architecture  as  model 

Need  operations/architecture  view  of  SoS 
that  individual  systems  must  plug  into- 
need  someone  responsible  for  this 


Prioritization  of  SoS  capabilities  at  high 
(OSD)  level  required  to  permit  constituent 
PM  to  manage  development  and  delivery. 
With  funding  at  SoS 

Measure  and  baseline  SoS  capability  thru 
T&E  w/o  requirements.  Where  do  we  get 
metrics? 

Must  have  an  “enforcer”  capability 
manager  -  carrots  and  sticks 
Measure  SoS  capabilities  when  changes  to 
SoS  Baseline 

CONOPs  vs  innovative  use  of  systems  in 
face  of  changing  threat 
Move  from  paper  to  4  dimensions  to 
capture  SoS  capabilities  requirements. 
Use  of  modeling  tools  of  SoS  components 
delivered  with  each  component  to 
communicate  requirements 
Capability  flow  down  to  systems,  demo 
meeting  systems  capability 


Issue  2 :  If  “requirements”  are  not  clearly  defined 

up  front  for  a  SoS,  what  is  basis  for  T&E  of  an 
SoS? 


Approach  to  addressing  issue 

•  The  DOD  needs  a  top-down  (architecture,  requirements, 
context,  expectation)  flow-process  to  systems  within  the  SoS 

•  Needs  authority  &  funding  to  enforce  capability  fulfillment 

•  Needs  to  be  flexible  enough  to  meet  changing  needs  and 
threats  and  CONOPS/operator  innovation. 

•  Determine  the  right  balance  between  system  test  to  sos-  test  to 
SOS  level  test 
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Issue  3  What  is  the  relationship  between  SoS  metrics 

and  T&E  objectives? 

Discussion 

•  SoS  T&E  is  focused  on  continuous  improvement  of  the  SoS  (as  compared 
to  system  T&E  which  is  focused  on  the  field,  fix,  or  don’t  field  decision) 

•  Continuous  SoS  T&E  requires 

—  Stable/consistent  metrics 

—  Consistent  approach  to  defining  evolving  baseline 

-  A  way  to  deal  with  emergent  behavior  (technical,  organization,  human)  -  positive  or 
negative 

-  Need  to  leverage  wide  range  of  opportunities  for  test  environments 

-  Continuous  improvement  means  continuous  testing  ;  Built  in  test  instrumentation  for 
feedback  from  field 

•  SoS  metrics 

-  Do  not  address  discrete  behaviors  of  systems  (as  do  system  metrics) 

-  Do  address  end  to  end  performance  across  systems  in  SoS  toward  capability  objectives  of 
the  SoS 

•  What  is  objective  of  T&E  for  an  SoS? 

-  Development  information  on  capabilities  and  limitations  of  SoS  to  inform  end  users  and 
ongoing  SoS  evolution  (as  compared  to  system  T&E  which  is  assessment  of  whether 
system  meets  requirements) 

•  SoS  T&E  customers? 

-  End  user  and  SoS  SE  team  (as  compared  to  system  T&E  where  aquisition  community  is 
the  customer) 

•  SoS  T&E  should  be  risk  driven:  focus  on  areas  of  risk  to  SoS  or  systems 
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Issue  3  What  is  the  relationship  between  SoS 

metrics  and  T&E  objectives? 


Approaches  to  addressing  issue 

•  Characterize  SoS  T&E  as  continuous  improvement,  document 
the  approach  and  share  with  the  community 

•  Radically  change  how  we  look  at  testing  given  the  growing 
prevalence  of  SoS 

—  Concepts  of  DT  and  OT  don’t  really  fit 

—  Inefficient  to  address  systems  in  operational  SoS 
environment  on  a  system  by  system  basis  (OT  today) 

—  Continue  to  test  individual  systems  to  assess  whether  we 
have  developed  what  we  asked  for 

—  Create  a  new  approach  to  OT,  by  cross  systems  support  for 
testing  capabilities 
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Issue  4 


Are  expected  cumulative  impacts  of  systems 
changes  on  SoS  performance  the  same  as  SoS 
performance  objectives? 

Discussion 


To  address  these  issues  you  need  to  fix 

-  Define  the  SoS  and  its  performance  objectives 

•  Constituent  systems  that  are  part  of  the  SoS 

•  Which  parts  of  the  constituents  contribute  to  the  SoS 
objectives 

-  Describe  the  current  and  future  state  of  the  changing 
systems  (Baselines) 

-  Assign  ownership  of  SoS  performance  objectives 

-  Big  challenge;  leadership  issue,  etc 

•  More  collaborative  approach  for  stakeholders  of  SoS  , 

Emergent  behavior  -  interaction  of  systems, 
humans,  system  and  organization  along  with 
constant  change  of  the  parts 

Bounds  of  human  impact 

-  Operator  -  leader  -  mission 

-  The  people  side  of  systems  « 

Training  and  development  of  the  evaluators 
(and  the  end  users) 

Expensive  to  assess  if  capabilities  are  realized 
(hard  to  do) 

-  Doing  more  with  less? 

-  Disconnect  thinking  and  reality? 


Leadership  understanding  of  SE  and  SoS 

-  Is  there  competency  to  make  decisions  and  know 
the  impact  and  implications? 

•  Trades  without  know  the  desired  outcome  can  be 
achieved 

-  Evaluation  on  an  SoS  basis  vs  individual 
systems  and  their  acquisitions 

-  Timing  and  who  benefits  (lack  of  rewards 
systems) 

-  Accountability  for  SoS 

Continued  improvement,  assessment, 
and  alignment  because  objectives  have 
changed 

-  More  data  from  fielded  systems 

Connections  to  fielded  side  of  the  house 
(doesn’t  deal  well  with  change) 

“Measurement  system’  for  system 

-  Analysis  of  impacts 

-  M&S? 

-  Risks;  “we  are  not  sure  but. . with  some 
mitigation 

-  Regression  testing  and  configuration  of  SoS 

-  Comparative  analysis 
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Issue  4  Are  expected  cumulative  impacts  of  systems 

changes  on  SoS  performance  the  same  as  SoS 
performance  objectives? 


Approaches  to  addressing  issue 

•  Influence  assigning  leadership  responsibility  and  ownership 
of  defined  SoS  capability  and  associate  performance 
objectives 

•  Establish  incentives  of  constituent  systems  to  collaborate  and 
achieve  SoS  performance  objectives 

•  Map  SoS  capabilities  and  performance  objectives  to 
constituent  systems  (under  configuration  control) 

•  Continual  assessment,  improvement,  and  realignment  is 
required  (incremental  approach)  focused  on  end  user) 

•  Create  a  guidance  framework  for  emergent  behaviors  of 
changing  to  be  measured  and  managed 
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Issue  5 


Discussion 


Are  expected  cumulative  impacts  of  systems  changes 

on  SoS  performance  the  same  as  SoS  performance 
objectives? 

How  do  you  test  the  contribution  of  a  system  to  the 
end  to  end  SoS  performance  in  the  absence  of  other 
SoS  elements  critical  to  the  SoS  results? 


•  Trying  to  assemble  all  piece  parts 
for  T&E 

•  So  many  variables  that  can  impact 
T&E  outcome 

•  Reliance  on  other  programs  (e.g., 
JTRS)  for  capabilities  that  can  slip 
in  schedule  or  are  never  delivered 

•  Spanning  “use-case”  space  with  a 
reasonable  set  of  resources  and 
schedule 

•  Need  defined  set  of  requirements 
(but,  of  course,  this  is  part  of  the 
problem  space) 

•  What  does  a  T&E  strategy  look 
like? 

•  How  account  for  “the  network” 
and  stresses  to  it? 


•  DoD  should  require  programs  to  share/  make 
transparent  to  other  programs  their 
development,  DT  and  other  data  (obstacles: 
proprietary/ security) 

•  Recommend  ways  to  systems  instrument  to 
enable  post-fielding  collection  of  “test”  data 

•  Operations,  exercises,  training 

•  DoD  should  develop  a  common  approach  to 
accounting  for  “the  network”  as  a  constituent 
of  all  SoSs  for  purposes  of  T&E 

•  DoD  articulate  purpose  of  SoS  T&E 

-  Is  it  a  capability  demo  (  “what  do  we  have?”) 

-  Is  it  a  classical  check  against  requirements? 

-  The  real  purpose  of  SoS  T&E  is  to  answer: 

•  Is  the  new  capability  operationally  useful  (whether  or 
not  it  “met”  requirements);  what  are  risks? 

-  How  can  the  new  capability  be  used? 

-  What  further  changes  are  required? 
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Issue  5  Are  expected  cumulative  impacts  of  systems  changes  on 

SoS  performance  the  same  as  SoS  performance  objectives? 

How  do  you  test  the  contribution  of  a  system  to  the  end  to 
end  SoS  performance  in  the  absence  of  other  SoS  elements 
critical  to  the  SoS  results? 

Approach 

•  M&S  of  piece  parts  that  are  not  yet  ready  to  be  tested  (but  issues 
between  M&S  for  individual  system  performance  versus  effects-based 
M&S)  -  potential  solution  to  issue  #1. 

•  Architectures  and  synchronizing  them  an  enabler  of  T&E  (provides 
well-defined  baseline;  can  measure  deltas  against  the  baseline) 

•  Combinatorial  test  &  design  (suggested  as  potential  solution  to  issue 

#2). 

•  Model-test-model  approach  suggested  for  way  to  accommodate 
emergent  behavior 

•  Field  exercises  -  instrumentation  to  collect  data 

•  Training  as  a  T&E  opportunity 

•  No  SoS  requirement  =>  no  TEMP  for  SoS  capabilities  =>  no  SoS  T&E 
funding.  Therefore  need  a  capability  (SoS)  focused,  cross-system, 
integrated  test  schedule  that  builds  to  a  graduation-level  event,  (some 
disagreement  re.  existence  of  such  an  event).  Push  SoS  T&E  to 
fleet/operators  as  proof  of  IOC  (need  fleet  experimentation  funding). 
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Test  Planning  -Advancing  the  Science 


NORTMROP  GRUMMAN 


•  Agenda 

-  Some  opening  thoughts 

-  Why  develop  a  Test  Plan? 

-  What  is  a  Test  Plan? 

-  What  do  you  plan? 

-  Where  does  a  Test  Plan’s  data  come  from  ? 

-  How  do  we  plan? 

•  Verification 

•  Safe  testing 

•  Test  Techniques 

•  Test  Tools 

•  Test  resources 

-  Keeping  it  all  straight 

-  Let’s  Plan 

-  Conclusions 


"Let  our  advance  worrying  become  advance  thinking  and  planning" 

Winston  Churchill 
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Test  Planning 


NORTHROP  GRUMMAN 


— Appaximately  20%-30%  of  the 
overall  projects  work  should  be 
allocated  to  testing.” 

— Bgardless  of  how  much  testing  is 
allocated  for  the  project,  it  is 
important  to  note  that  acceptable 
test  results  do  not  necessarily 
require  perfection.  Acceptable 
testing  is  more  about  validating 
what  is  agreed  to  be  done  rather 
than  being  perfect  or  even 
exceeding  expectations.” 

-  Harold  Kerzner’s  book  Project  Management  a  Systems 
Approach  to  Planning,  Scheduling,  and  Controlling  John 
Wiley  &  Sons,  Inc 


Stay  On  Target 
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Remember  This? 


NORTHROP  GRUMMAN 


Number 

Effective 

And 

Suitable 


Cumulative  IOT&E  Results  Through 

FY  2008 


NDIA  11th  Annual  System  Engineering  Conference  Proceedings,  Keynote  Presentation,  HON  Charles  McQueary,  Director,  Operational  Test  & 
Evaluation  -  October  2008 
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Tutorial  Description 


NORTMROP  GRUMMAN 


•  From  verification  to  test  plan  modeling  and  test  plan  generation, 
participants  will  see  the  processes  and  tool  sets  in  action. 

•  To  demonstrate  some  of  these  capabilities,  participants  will  generate 
test  requirements  and  objectives,  model  the  plan,  optimize  the  plan 
and  assign  resources,  and  finally  generate  a  simple  test  plan  while 
maintaining  connections  to  the  original  requirements  intent. 


•  Fools  rush  in 
Where  wise  men  never  go. 
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Test  Planning  -  Advancing  the  Science 


NORTMROP  GRUMMAN 


Tutorial  Style 


INTERACTIVE 


IDEA  SHARING 


TEAM  EXERCISER 

■  - 1  _  1  V 
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Why  Develop  a 
Test  Plan  ? 
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How  Do  Plans  Help  The  Program? 


NORTMROP  GRUMMAN 


•  Identifies  the  test  program  and  test  program  resources 

•  Provides  a  method  to  manage  the  test  program 

•  Optimized  test  plan  saves  program  cost 

•  Ensures  the  test  program  is  traceable  to  the  product  architecture 
(requirements) 

•  Test  plan  can  help  manage  program  changes 

•  Test  plans  foster  communications 
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Advancing  the  Science 


NORTMROP  GRUMMAN 


•  Test  planning  typically  relies  on 

-  Experience 

-  Requirements 

-  DWWDLT  (Did  What  We  Did  Last  Time) 

-  Lessons  learned 

-  Working  teams  /  meetings 

-  Schedules 

•  Test  planning  must  advance  using 

-  Experience 

-  Doing  what  is  required  (optimizing  the  test  program) 

-  Working  teams  /  meetings 

-  Schedules 

-  Test  plan  modeling  (utilizing  SE  based  tool  set) 

-  Appropriate  application  of  design  of  experiments 

-  Collaborative  techniques  and  tools  to  encompass  the  entire  programs  test  program 

-  Support  rapid  evaluation  based  on  programmatic  changes 
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Why  Do  We  Plan? 


NORTHROP  GRUMMAN 


•Planning  allows  one  to  stay  on 
target,  project  the  future,  and 
assess  the  impact  of  change. 

•Planning  identifies  problems  and 
points  the  way  to  solutions.  J  ust 
taking  a  systematic,  thorough  look 
at  the  current  situation  and  thinking 
about  the  implications  for  the 
future,  can  bring  these  things  to 
light. 

•It  helps  us  to  do  first  things  first. 

In  other  words,  it  provides  a 
rationale  for  assigning  priorities. 

•A  good  plan  will  suggest  answers 
to  perplexing  questions. 


Planning  is  "intelligent  cooperation  with  the  inevitable." 
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The  TEMP  Lifecycle  Value 


NORTMROP  GRUMMAN 


•  The  TEMP  identifies  and  integrates  all  of  the  T&E  requirements  with 
the  program's  acquisition  strategy  and  requirements.  The  temps  for 
OSD  oversight  programs  follow  the  DAG  format  and  must  be  approved 
by  the  director,  DT&E  and  the  director,  OT&E.  Service  approved  temps 
are  developed  according  to  service  regulations  and  guidance.  The 
TEMP  is  used  by  the  program  office  to: 

-  Provide  an  overall  test  management  plan  within  the  acquisition  strategy  bounds, 

-  Identify  overall  T&E  activities  by  the  government  and  system  contractor, 

-  Guide  the  development  of  specific  test  events  and  integration  of  detailed  test  plans 
for  those  activities  by  summarizing  relevant  performance  requirements,  and 

-  Document  T&E  schedule  and  resource  requirements  Acouipedia- 

Defense  Acquisition  University  Web  Site 


Test  Planning  is  a  Lifecycle  Event  -  Programs  Must  Not  Dismiss  the  Test  Plan  Importance 


11 


Cleared  for  Public  Release  11-0188  Stephen  Scukanec  Northrop  Grumman  3/14/11 


What  is  a  Test 
Plan? 
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What  Is  A  Test  Plan? 


NORTMROP  GRUMMAN 


Microsoft 


□ 


Create  a  test  plan 


Plan  the  test  lab 


Design  the  test  lab 


Develop  the 
test  lab 


Design  test  cases 


Conduct  tests 


Jse  the  lab  after 
deployment 


Define  testing 
scope  and 
objectives 

Define  testing 
methodology 

Identify  required 
resources 

"Identify  the 
features  and 
functions  to  test 


Identify  risk 
factors 


Establish  a 
testing  schedule 


Early  in  the  deployment 
planning  phase,  the  testing 
team  creates  a  test  plan.  The 

test  plan  defines  the 
objectives  and  scope  of  the 
testing  effort,  and  identifies 
the  methodology  that  your 
team  will  use  to  conduct 
tests.  It  also  identifies  the 
hardware,  software,  and 
tools  required  for  testing  and 
the  features  and  functions  that 
will  be  tested.  A  well-rounded 
test  plan  notes  any  risk 
factors  that  jeopardize 
testing  and  includes  a 
testing  schedule. 


TABLE  OF  CONTENTS 
TABLE  OF  CONTENTS 
DOCUMENT  INFORMATION 
LAB  TEST  PARTICIPANTS 
REVISION  HISTORY 
CONTENTS 

EXECUTIVE  SUMMARY 
TEST  SCOPE 
LAB  TEST  GOALS 

SUCCESS  CRITERIA  (OBJECTIVES)  / 

CRITICAL  METRICS 

TEST  TOOLS 

ASSUMPTIONS 

RISK  FACTORS 

BRIEF  HISTORY  OF  THE  ITEM  BEING 

TESTED 

USE  CASES 

NOT  IN  TEST  SCOPE 


Microsoft  Tech  Net 

http://technet.microsoft.com/en-us/library/cc781 572(WS.  10).  aspx  http://technet.microsoft.com/en-us/library/ee221 154(EXCHG. 80). aspx 
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What  Is  A  Test  Plan? 


NORTHROP  GRUMMAN 


M 


CDC  Unified  Process 
Practices  Guide 

Test  Planning  Practices  Guide 


Ar  reaardless  of 

nd  approaches.  However.  reg 

There  are  a  ***' ^^ess  Th.s  stage  o? the 

approach  ts  uSe '  _L„  ,  ,hf>  test  planning  Pr<?c"  „  Deriorm  those  tests 

'  & 
°  Test?ngSapP^^^ef5^^ng^ha^°^''^gS\Scotnp^e^d'wSst 

o  Environmental  nee^  needs 

0  staffing  and  trammy 


Center  for  Disease  Control 
2007 

http://www2.cdc.gov/cdcup/library/practices_guides/CDC_UP_Test_Planning 

_Practices_Guide.pdf 


INTRODUCTION 

Purpose  of  The  Test  Plan  Document 
Compatibility  testing 

Test  Risks  /  Issues 

Items  to  be  Tested /Not  Tested 

Test  Approach(s) 

Test  Regulatory  /  Mandate  Criteria 
Test  Pass  /  Fail  Criteria 
Test  Entry  /  Exit  Criteria 
Test  Deliverables 

Test  Suspension  /  Resumption  Criteria 
Test  Environmental  /  Staffing  /  Training  Needs 

Conformance  Testing 
Functional  Testing 
Load  Testing 
Performance  Testing 
Regression  Testing 
Stress  Testing 
System  Testing 
Unit  Testing 

User  Acceptance  Testing 
Test  Plan  Approval 
Appendix  A:  References 
Appendix  B:  Key  Terms 

Each  subsection  is  repeated  in  each  major 
section 


www2.cdc.gov/cdcup/library/templates/CDC_UP_Test_Plan_Template.doc 
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What  Is  A  Test  Plan? 


NORTHROP  GRUMMAN 


€The  TEMP  describes  the  acquisition  program's  planned  T&E  over  the  program's  life  cycle  and  identifies  evaluation 
criteria  for  the  testers.  It  serves  as  an  executive  summary  of  the  overall  test  program.  Building  on  the  foundations 
laid  in  the  TES,  the  TEMP  identifies  and  integrates  all  of  the  T&E  requirements  with  the  program's  Acquisition 
Strategy  and  requirements 


DOD  /  Air  Force  TEMP  TOC  -  2  Levels  Deep 


PART  1  -  INTRODUCTION 

1.1PURP0SE 

1.2MISSION  DESCRIPTION 
1.3SYSTEM  DESCRIPTION 

PART  III  -  TEST  AND  EVALUATION  STRATEGY 

3.1T&E  STRATEGY 

3.2  EVALUATION  FRAMEWORK 

Figure  3.1  -  Top-Level  Evaluation  Framework  Matrix 

3.3  Developmental  Evaluation  Approach 

3.4  Live  Fire  Evaluation  Approach 

3.5  Certification  for  IOT&E 

3.6  Operational  Evaluation  Approach 

3.7  OTHER  CERTIFICATIONS 

3.8  RELIABILITY  GROWTH 

3.9  FUTURE  TEST  AND  EVALUATION 


PART  II  -  TEST  PROGRAM  MANAGEMENT  AND  SCHEDULE 

2.0  T&E  MANAGEMENT 
2.1.1T&E  Organizational  Construct 
2.2Common  T&E  Data  Base  Requirements 
2.3DEFICIENCY  REPORTING 
2.4  TEMP  UPDATES 

2.51  NTEGRATED  TEST  PROGRAM  SCHEDULE 
Figure  2.1  -  Integrated  Test  Program  Schedule 


PART  IV  -  RESOURCE  SUMMARY 

4.1  Introduction 
4.1.1Test  Articles 

4.1.2Test  Sites  and  Instrumentation 
4.1.3Test  Support  Equipment 
4.1.4Threat  Representation 
4.1.5  Test  Targets  and  Expendables 
4.1.60perational  Force  Test  Support 
4.1.7Models,  Simulations,  and  Test-beds 
4.1.8J  oint  Operational  Test  Environment 
4.1.9Special  Requirements 

4.2  Federal,  State,  Local  Requirements 

4.3  Manpower/Personnel  Training 

4.4  Test  Funding  Summary 

Table  4.1  Resource  Summary  Matrix 
APPENDIX  A  -  BIBLIOGRAPHY 
APPENDIX  B- ACRONYMS 
APPENDIX  C  -  POINTS  OF  CONTACT 
ADDITIONAL  APPENDICES  AS  NEEDED 
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What  do  We 
Plan? 
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What  Do  We  Plan  ? 


NORTHROP  GRUMMAN 


•  Tenet 

©a  widely  held  belief;  especially:  one  held  in  common  by  members 
of  a  group  or  profession 


•  Feature 

©a  part  or  detail  that  stands  out 


Define  the  tenets  and  features  of  a  good  test  plan. 


Breakjnto  Team 
•Select  a  Spokesperson 
•Develop  3-4  Key  Tenets  Of  a  Good  Test 
Plan 

•Develop  3-4  Key  Features  of  a  Test  Plan 
•Write  Them  Down 
•Share  With  Community 


17 


Cleared  for  Public  Release  11-0188  Stephen  Scukanec  Northrop  Grumman  3/14/11 


What  Do  We  Plan  ? 


NORTHROP  GRUMMAN 


•  Tenets  of  a  Good  Test  Plan 

-  Defines  Test  Strategy 

-  Establishes  Test  Program  Management 

-  Documents  the  Test  Program 

-  Identifies  the  Needed  Resources 

•  Features  of  a  Good  Test  Plan 

-  Can  be  used  to  manage  the  test  program 
lifecycle 

-  Covers  all  program  level  test  responsibilities 

-  Traceable 

-  Adjustable 

-  Is  used  as  the  requirements  document  for  test 
procedures 

-  Avoids  obsolescence 


The  Test  Plan  is  the  Test  Procedure's  Requirement  Document 


Cleared  for  Public  Release  11-0188  Stephen  Scukanec  Northrop  Grumman  3/14/11 


Tenet  Characteristics  -  Strategy 
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The  skill  of  making  or  carrying  out  plans  to 

achieve  a  goal 
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The  Approach 
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Tenet  Characteristics  -  Management 


NORTMROP  GRUMMAN 


Judicious  use  of  means  to  accomplish  an  end 
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Test  Program  Controls 
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Tenet  Characteristics  -  Program 


NORTHROP  GRUMMAN 


A  planned,  coordinated  group  of  activities, 
procedures,  etc.,  often  for  a  specific  purpose,  or  a 


Mcniaixt- 

A/\fchster. 


facility  offering  such  a  series  of  activities 


Pr°gram 
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Detailed  Program  Activities 
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Tenet  Characteristics  -  Resources 
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A  source  of  supply,  support,  or  aid,  esp.  one 


Mcrrbm- 
.  Webster. 


can  be  readily  drawn  upon  when  needed 


that 


Test  Facilitators 
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Tenet  Alignment  ? 
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Tenet 

Microsoft 

CDC 

DoD 

Strategy 

Test  scope 

Lab  test  goals 

Not  in  test  scope 

Test  Approach(s)  Items 
to  be  Tested  /  Not 

Tested 

Part  1&2 

> 

Management 

Metrics 

Schedule 

(Embedded) 

Test  Entry  /  Exit  Criteria 
Test  Deliverables 

Approval 

Schedule  (embedded) 

Part  3 

Program 

Success  criteria 
(objectives)  / 
critical  metrics 
risk  factors 

Use  cases 

Test  Pass  /  Fail  Criteria 
Test  Risks  /  Issues 

Part  2 

> 

Resource 

Lab  test 
participants 

Test  tools 

Test  Environmental  / 
Staffing  /  Training  Needs 

Part  4 
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Test  Plans  Have  Common  Tenets  Across  Much  of  Industry  and  Government 
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Test  Plan  Data 
Sources 
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A  Excerpt  From  The  DAU  T&E  Course 
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•  Test  personnel  must  keep  in  mind  that  system  test  and  evaluation  is 
not  limited  to  the  technical  performance  of  hardware  and  software. 

•  Evaluation  of  a  complete  system  can  include  a  wide  range  of  factors, 
such  as  requirements,  support  requirements,  arming  distance,  and 
weight. 

•  Evaluation  of  a  complete  system  must  include  a  wide  range  of  factors 
in  additional  to  purely  technical  ones,  such  as:  training  and  human 
factor  requirements,  supportability  and  maintainability,  facilities,  etc. 


DAU  Fundamentals  of  Test  and  Evaluation  Course  Tst  102-  Evaluation  Considerations 
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Test  Plan  Input  Sources 
Military  Program 


NORTHROP  GRUMMAN 


I  Source 

1  Owner 

I  Characteristic 

)  Product 

iTest  Plan  Input 

Operational  Need 

Sponsor 

COIC,  CTPs, 

Objectives  & 

Thresholds 

CDD,  Evaluation 

Criteria 

Measures  of 

Effectiveness 

Key  Performance 

Parameters 

Test  Strategy 

Sponsor / 
Contractor 

Environment 

TES/TEMP/User 

Test  Conditions 

Resources 

Requirements 

SE 

Compliance  Criteria  / 
Methodology 

Specifications 

Verification  Criteria 

Policies 

Government  / 
Sponsor 

Agencies/ 

Contractor 

Environmental 

Concerns 

Accepted  Approaches 

Policies 

Standards 

Directives 

SEP 

Accepted  Test  Approaches 

Test  Experience 

T&E 

Safe  &  effective  test 
techniques 

Lessons  Learned, 
Previous  Program 
Documentation 

Test  Techniques 

Tech  Maturation 
Plan 

Design/  Eng 

Risk 

Opportunities 

Tech  Maturation 

Tech  Maturation  Plan 
Risk  Plan 

Design  Development  Test 
Requirements 

Manufacturing 

Strategy 

Manufacturing 

Acceptance  Criteria 

Manufacturing  Plan 

Testability  Requirements 

Tools 
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When  Do  You 
Test  Plan? 
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Does  any  of  this  Sound  Familiar? 


NORTMROP  GRUMMAN 


How  do  You  Plan  for  Testing? 

-  We  plan  as  we  go 

-  Our  UP  /  Master  Test  Plan  is 
Strategic  Guide  for  the  Test 
Program,  we  develop  lower  tiered 
plans  for  the  detailed  facility 
dependant  tasks 

-  We  don’t  write  an  ITP  we  rely  on 
the  lowered  tier  plans 


When  do  you  Plan  for  Testing? 

-  We  build  a  strategic  plan  for  early 
program  milestones  (PDR  or  later) 

Our  lower  tier  plans  are  developed 
before  the  test  TRR 

-  Once  the  lower  tier  plans  are 
developed  we  rely  on  the  test 
procedures  to  adjust  the  plan  as 
necessary 


-  We  write  only  what  is  necessary  to 
get  through  the  milestone  delivery 

-  The  ITP  /  MTP  is  valid  until  CDR  or 
it’s  equivalent 

-  We  use  our  program  schedule  as 
our  test  plan 
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When  Do  We  Plan 


NORTMROP  GRUMMAN 


Materiel 
Develoi 
D*£f< 


MS  A 


MS  5 


MS  C 


JC1DS  Process 


"Pnst-PDR PoaKCDR 
Assessment  ^■‘vKssessment 

PDR  CDR 


RateJ?rcrouct1on 
Decision  Review 


Break  Into  Teams 

•Select  a  spokesperson 

•Establish  key  test  planning  inputs  by  program 

phase 

•Establish  phases  for  TES,  TEMP,  contractor  plan, 
DT&E  and  OT&E  plans 

•Decide  the  state  (draft,  1st  release,  final,  etc;)  of 
the  test  strategy  /  plan  at  each  program  milestone 
•Ms-A,  MS-B,  MS-C,  PDR,  CDR,  etc; 

•Write  them  down 
Share  with  community 
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Test  Planning-  A  Lifecycle  Look 


NORTHROP  GRUMMAN 


Materiel 

Development 

Decision 


MS  A 

▲ 


MS  B 


CBA 

ICD 

Materiel 

Technology 

I-  J 

li 

pH 

ftji  r\  -  oOlUIlOn 

Analysis 

Development 

IOC 

k 


Engineering  and 
Manufacturing 
Development 


CPD 


Production  and 
Deployment 


O&S 


FOT&E- 
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-\^r  VI 

\  •  \I 

\ _ ^  ' 

T&E 

•“....incorporation  of  T&E  considerations  and  requirements  begins 
at  the  onset  of  program  planning  during  the  Material  Solutions 

Analysis  and  TD  phases”  (paragraph  2.1  Incorporating  Test  and  Evaluation  into 

Department  of  Defense  Acquisition  Contracts) 

|  Intent  must  1 

oe  maintained  throughout  the  pro 

grams  lifecycle  to  ensure  warfighter  need  is  provided 
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Test  Planning  -  Advancing  the  Science 


NORTHROP  GRUMMAN 


•  Test  planning  starts  at  program  inception 

•  Test  planning  support  the  development  of  product  architecture  and 
requirements 

•  Test  planning  requires  the  proper  skill  mix  with  lifecycle  experience 

•  Test  planning  is  a  lifecycle  task 

•  Test  planning  requires  a  collaborative,  program  integrated,  model  based 
tool  set. 

•  Test  planning  should  look  front  to  back  and  not  back  to  front 

•  Test  planning  should  help  decide  the  test  techniques,  not  the  other  way 
around. 

-  Just  because  you  used  a  laboratory  last  time  doesn’t  mean  you  need  it  this  time. 
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How  Do  We 
Plan  ? 
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How  Do  We  Plan 


NORTMROP  GRUMMAN 


Pick  a  Planning  Method(s) 

Pick  a  Planning  Tool(s) 

Apply  Experience 

Get  Lessons  Learned  and  Other 
Program  Experience 

Get  User  Input 


Understand  the  Available  Test 
Techniques 

Understand  the  Verification 
Needs 

Learn  the  Policies 

Determine  the  Sequences  and 
Prerequisites 

Write  it  all  Down 
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How  Do  We  Plan? 


NORTMROP  GRUMMAN 


•  Consider  an  improved  air-to-air  missile  system  that  requires  testing — 
Missile  A  Improved.  Suppose  the  original  Missile  A  had  an  historical  hit 
rate  of  70%.  The  test  design  must  evaluate  whether  the  improved 
missile  is  at  least  equal  to  or  better  than  the  original  in  — tangt  hit” 
success.  How  many  shots  do  we  need  to  make  to  determine  the 
performance  of  the  improved  Missile  A? 

•  Starting  with  a  blank  sheet  of  paper,  the  test  engineer  must  define  the 
appropriate  number.  But  what  is  the  number  of  shots  necessary  to 
verify  the  improved  Missile  A.  Maybe  the  number  is  3,  because  that  is 
what  the  available  time  or  money  will  support.  Maybe  the  number  is  8 
because  the  engineer  just  likes  8.  Maybe  the  number  is  10  because 
the  engineer  is  challenged  by  fractions.  Or  maybe  the  number  is  30 
because  in  life  something  good  happens  at  30!  There  is  no  statistical 
backing  for  any  of  these  numbers,  but  all  remain  possibilities.  For  no 
particular  reason,  the  engineer  chooses  10. 

Design  of  Experiments  Applied  to  Flight  Testing  -  Leslie  L.  Bordelon 
U.  S.  Air  Force  Senior  Executive  Service  Retired  -  RTO-EN-SCI-176 
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Traditional  Test  Planning  Methods 
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•  Test  Team  Planning  Approach 

-  Intuition  -  SME  opinions,  Quick  and  Easy,  Not  Much  Detailed  Planning  Required 

-  Do  What  We  Did  Last  Time  (DWWDLT)  -  Defined  Trade  Space,  Cost  and 
Schedule,  May  Not  Examine  New  Capabilities  Under  Changed  Environment 

-  One  Factor  at  a  Time  (OFAT)  -  Organized,  repeatable,  Non-interactive 

-  Best  Guess  -  Cost  and  Schedule  Driven 

-  Use  Comparable  Data  -  Adds  Supporting  Data,  Lacks  Fidelity  to  New  Case 


During  the  1920s,  a  British  statistician 
named  Ronald  Fisher  put  the  finishing 
touches  on  a  method  for  making 
breakthrough  discoveries.  Some  70  years 
later,  Fisher's  method,  now  known  as 
design  of  experiments,  has  become  a 
powerful  tool  for  engineers  and 
researchers. 


Design  of 
Experiments  ? 


35 


Cleared  for  Public  Release  11-0188  Stephen  Scukanec  Northrop  Grumman  3/14/11 


Design  Of  Experiments 


NORTHROP  GRUMMAN 


"The  53d  Wing  (53  WG)  of  Air  Combat  Command  (ACC)  at  Eglin  Air  Force  Base  (AFB),  Florida, 
has  used  experimental  design  on  over  25  operations  in  the  past  14  years." 

Design  of  Experiments  Applied  to  Flight  Testing  Leslie  L.  Bordelon  U.  S.  Air  Force  Senior  Executive  Service  Retired  RTO-EN-SCI-176 


"As  I  review  Test  and  Evaluation  Master 
Plans  (TEMPs)  and  Test  Plans,  I  am  looking 
for  specific  information.  In  general,  I  am 
looking  for  substance  vice  a  'cookbook'  or 
template  approach  -each  program  is  unique 
and  will  require  thoughtful  tradeoffs  in  how 
this  guidance  is  applied.  A  "designed" 
experiment  is  a  test  or  test  program,  planned 
specifically  to  determine  the  effect  of  a  factor 
or  several  factors  (also  called  independent 
variables)  on  one  or  more  measured 
responses  (also  called  dependent  variables)." 

Guidance  on  the  use  of  Design  of  Experiments  (DOE)  in  Operational  Test  and  Evaluation 
J .  Michael  Gilmore  Director  OT&E  10-19-2010 


Use  of  Design  of  Experiments  to  Determine  the  Critical  Technical  Parameters 

and  Evaluation  Framework  in  the  T&E  Strategy  Darleen  Mosser-Kerner,  Mickey  Quintrall 

26th  Annual  National  T&E  Conference  March  2nd  2010 


Conclusion 


•  Some  experts  contend  that  if  you  are  developing 
a  new  product  or  process,  it  is  not  the  right  time 
for  DOE. 

•  Best  suited  for  where  interactions  and  effects 
among  a  handful  of  strictly  controlled  variables 
are  of  interest.  In  experiments  with  higher 
complexity,  the  number  of  variables  must  be 
reduced,  which  can  result  in  omission  of  many 
of  the  possible  combinations  of  interactions. 


DT&E  needs  to  investigate  further,  prior  to  issuing  OSD 
TES  and  TEMP  guidance  on  implementing  DOE  across  the 
Acquisition  Lifecycle 

[  UNCLASSIFIED 

DOE  if  J  udicially  Applied  Can  Aid  in  Test  Planning  Decisions 
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Verification 
Is  this  Really 
Test? 
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Verification  Requirements 
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How  Do  you  Write  One? 

Answer  the  Following  Questions: 

1 .  Objective  -  What  is  the  purpose  of 
this  verification? 

2.  Method  -  What  method  do  you  need 
performed?  What  are  the 
verification  circumstances  (e.g., 
laboratory,  desk-top  analysis,  flight 
test)? 

3.  Environment  -  What  are  the 
environmental  conditions  under 
which  the  item  will  be  verified? 

4.  Special  Conditions  (if  necessary)  - 
Are  there  any  unique  conditions 
(e.g.,  item  configurations)  necessary 
for  the  execution  of  the  verification? 

5.  Success  Criteria  -  What  results  are 
to  expected? 


Early  Test  Planning  Starts  with  Requirements  Development 


Why  are  they  Needed? 

-  Verification  requirements  specify  the 
verification  events  needed  to  prove 
the  satisfaction  of  the  product 
requirements  and  help  to  define  the 
verification  process  and  environment 

-  Verification  requirements  are 
necessary  for  at  least  two  reasons: 

•  Existence  of  verification 
requirements  demonstrates 
verifiability  of  product  requirements 

•  Agreed-to  verification  requirements 
define  the  verification  program  by 
which  the  contractor  shows  that  the 
product  is  what  the  customer 
contracted  for. 
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Verification  Requirement  Example 
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•  Requirement 

-  The  product  shall  provide  a  communications  system  (defined  in  Figure  1) 
capable  of  communicating  with  the  recovery  forces  pre-  and  post-  landing 
with  both  audio  and  digital  communications. 


•  Verification  Statement  (1  of  3) 

-  Prove  that  the  product’s  communications  system  is  capable  of 

communicating  with  the  ground  command  team  by  performing  an  laboratory 
within  an  integrated  hardware/software  environment.  Testing  will  be 
conducted  with  the  system  operating  under  induced  interference  patterns  as 
defined  in  Figure  7.  Testing  will  show  that  the  product  can  transmit  and 
receive  to  standard  ground  recovery  forces  audio  at  frequencies  represented 
by  communications  devices  operating  in  the  VHF/AM  and  S  Band  Frequency 
Bands.  Voice  communications  will  be  measured  using  the  Perceptual 
evaluation  of  speech  quality  (PESQ)  P.862  defined  method.  Digital 
Communications  will  be  measured  by  ensuring  proper  communications  can 
be  established  by  the  receiving  unit. 
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Verification 


NORTMROP  GRUMMAN 


•  The  verification  program  is: 

-  Proof  that  the  sponsor  gets  what  they  asked  for 

-  The  collection  of  the  data  set  which  aids  in  the  compliance  assessment  of  a  design 
requirement 

-  The  collection  of  data  which  aids  in  the  assessment  that  A  program  has  fulfilled  its 
commitments 


The  main  purposes 


S' 


The  DoD  Engineering  Process 


Rda 

Chief 

SYSTEMS 

ENGINEER 


CONOPS 

Operational 
Needs 

Mission 

Operational 

Requirements 


Operations 
&  Maintenance 

Deployment 


Validation 


^  PLATFORM 

Platform 

9 

TESTING  (DT  JOT) 

•  Platform  Operation  Testing  (OT) 

A  5SSS:",""’s  /pixnwer 

S©«  Developmental  Ten 

\ 

SYSTEM 

System 

Functional 

Behavior 

9 

1  Jfc 

Verifies  ti  on  system 

ICD/CCD/CPO  SYSTEM 

/ 

1 

□  COMPONENTS 

\  Safe  Safe 

»  •  'n»ertec«  Management  COMPONENTS 

Implementation  A'“'y,l*P,oc,“  J 
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NDIA  Systems  Engineering  Strategic  Planning  System  Engineering  Challenges 
in  Naval  System  of  Systems  Ms.  Helene  Anderson 
Office  of  ASN  RDA  CHSENG  8  December  2010  Miami,  FL 
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Safe  Testing 
Techniques 
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Safe  Testing  Considerations 
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•  Apply  appropriate  test  methodology 

-  Pyramid,  bottoms  up,  agile,  regression 

•  Establish  prerequisites  to  safe  testing  (people  and  product) 

-  Hazardous  material  handling,  personnel  considerations,  test  point  /  envelope  expansion,  etc. 

•  Understand  and  comply  with  policies  which  effect  test  program  plan 

-  Ex;  test  range  requirements  ,FAA  policies,  space  qualification  standards 

•  Understand  constraints 

-  Test  limits,  data  limits,  environmental  conditions 

•  Establish  test  rules  and  entry  /  exit  criteria 

-  Know  when  you  have  completed  the  test,  know  when  you  have  good  data 

•  Establish  controls 

-  Security,  flight  line  policies,  configuration  management,  equipment  handling,  software 


Safe  and  Effective  Testing  a  Mandate  of  Every  Test  Program 
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Test  Technique  Examples 
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•  Conduct  a  low  speed  taxi  test 

-  Evaluate  Steering 

•  Ensure  aircraft  travels  down  the  runway  (+/-  5  feet  of  center)  at  speeds  up  to 
and  including  50  Knots. 

-  Evaluate  Communications 

•  Ensure  aircraft  communications  with  ATC  and  Ground  Station.  Ensure  no 
communication  drop  outs 
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A  Controversy 
Do  we  or  don’t 
we? 
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Testing  Techniques  Drives  Product 
Requirements 


NORTHROP  GRUMMAN 


•  The  Test  Plan  can  and  often  does  drive 
product  requirements 

-  Flight  termination  system 

-  Instrumentation 

-  Weight 

-  Power 

-  Space  /  volume 

-  Communications  protocol 

-  Frequency  allocations 

-  Others? 


CONOPS 

Operational 

Needs 


ICD  /  CDD 


SDS 


•  T&E  does  generate  requirements 

-  Identify  requirements  early  to  avoid  design 
impacts 

•  Don’t  be  late  to  need 


Contractor 

System 

Specification 

SRR 


Contractor 
Test  Plan 
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Test  Tools 
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Test  Planning  Tools 


NORTHROP  GRUMMAN 
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•  What  is  the  T&E  Test  Planning  Tool  Kit? 


Planning 

Test  Design 

Metrics 

1  1 

Assumptions 

Control  A 

Control  O  Wm 
Control  V  fm  if 

l£_jy 

[if1!} 

If  you  keep  doing  what  you're  doing 

You'll  keep  getting  what  you're  getting! 

Complete  Model  Based 
Test  Planning  -  Re-plan 
Test  Planning  Streamlining 
Costing  /  Scheduling 
Auto  Test  Plan 
Documentation 

Event  Planning 

Test  Design 

Test 

Plan  Validation 

Test  Program 
Work  Flow  /  Metrics 

Test  Planning 
Assumption 
Corroboration 
Apply  and  maintain 
Lessons  Learned 
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Test  Planning  -vs.  -  Test  Techniques 
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FLIGHT  TEST 


LABORATORIES 


TEST  RANGES 


Test  Planning  Defines  the  Test  Program  Test  Tools 
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Why  These  Techniques?  -  Some  Examples 
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LABORATORIES 


Off  Nominal 
Initial  Integration 
Interface  Development 
Problem  Resolution 
Functional  Checkout 


TEST  RANGES 


Installed  Performance  -  Static 
External  Interface  -  Operational 
Fit  Checks 

Low  Speed  Dynamics 
Initial  System  Control 
External  Communications 


FLYING  TEST 
BEDS 


Dynamic  Integration 

Dynamic  Functional  Design  Development 
High  Risk  Safety  Activities 
TRL  development  in  Operational 
Environment 

Targeted  Off-Nominal  Tests 


Operational  Environment 
Operational  Performance 


FLIGHT  TEST 


Pick  the  Right  Tools  for  the  Right  J  ob 
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Test  Resources 
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Test  Tools  -  Resources 
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•  Name  your  Resources  -  How  Many  -  How  Long 


Test  Article 


Instrumentation 
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Test  Tools  -  Resources 


NORTMROP  GRUMMAN 


•  Resources 

-  Come  in  all  varieties  (facilities,  equipment,  people,  tools) 

-  Effect  test  execution 

-  Have  changing  availability 

-  Are  required  for  test  execution 

-  Drive  schedule 

-  Drive  cost 

•  Test  Activities  are  Resource  Dependant 

-  There  can  be  many  resources  required  to  execute  a  test 

•  Think  a  SoS  test  activity 

-  Resources  can  get  lost  in  the  change  process 


Test  Planning  Must  Consider  the  Effect  Of  Resources  at  All  Times 


Cleared  for  Public  Release  11-0188  Stephen  Scukanec  Northrop  Grumman  3/14/11 


Putting  it  All  Together 
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Step 

1 


(•Start  the 
Program 
Right 


•Integrated 

Testing 


•Develop  Design 
Requirements 
•Develop  Test  Unique 
Requirements 
•Analyze  Architecture 
•Develop  Verification 
Requirements 


Test  Planning 


•Establish  Test  Model 
•Establish  Realistic  Test 
Environment 

•Build  Integrated  Test  Team 
•Develop  Integrated  Test 


\ 


•Does  the  Product  Meet  the 
Design  Criteria 
•Has  the  Evaluations  Been 
Conducted  in  a  Realistic 
Environment 
•Maintain  Procedure 
requirements 


Requirements 

Program 

)  U| 

Product 

Development 

j  % 

Verification 

•Operational  Assessments 
•Warfighter  Use 
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The  Test  and  Evaluation  Skill  mix  is  needed  to  help  a  program  start  on  the  right  foot 

Effective  Architecture 
Solid  Requirements 
Verifiable  Requirements 
Initiate  the  DoD  Integrated  Test  Program 
Ensure  the  Design  of  a  Realistic  Test  Program 
Initiate  Test  Strategy  and  Integrated  Planning 
aligned  with  Program  Risk,  Technology 
Development,  EMD  and  Production 
Determine  Long  Lead  and  Facilities  Needs 
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Keeping  it  All 
Straight 
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The  Test  Planning  Variables 


NORTHROP  GRUMMAN 


Collaborative 

Integrated 

T  raceable 

Schedule 

Tools 

Resources 

Risk 


Techniques 


Adaptable 


Managed 


Sequences 


AND  NOW  YOU 

Dependencies  WANT  TO  KNOW  THE 

PROGRAM  IMPACT  TO  A  CHANGE? 


Facilities 
Verification 
Lifecycle  Activities 
MoEs,  KPPs 
Realistic  Environments 
Operationally  Relevant 
Deliverable 
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NORTHROP  GRUMMAN 
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Test  Planning  Tools  -  Minimum  Requirements 
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•  Need 

-  Collaborative 

-  Handle  traceability 

-  Can  model  the  test  plan 

-  Support  test  optimization 

-  Connected  to  requirements  and  architecture 

-  Supports  the  verification  and  test  planning  criteria 

-  Can  produce  test  planning  artifacts 

-  Can  provide  configuration  management 

-  Flexible  to  adapt  to  program  needs 

-  Can  show  the  — big  pfcure” 

-  Can  be  used  by  all  program  personnel  -  all  skill  mixes 
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Let’s  Plan 


NORTMROP  GRUMMAN 


•  Review  OV-1 

•  Review  Requirements 

-  Provide  requirements  assessment  for 
requirements  2.5,  2.6,  2.7 

-  Add  verification  requirement 

-  Develop  verification  requirement  for 
requirement  2.3.2 


Review  Hierarchy 
Add  Resources 
Connect  flight  test  resources 
Optimize 


Add  traceability  for 
requirement  2.3.2 

Develop  test  activities 

-  Add  flight  test  phase,  (procedure 
development,  test  execution,  report) 

-  Connect  appropriate  verification 
requirements  to  test  activities  (2.3.1  .C, 
procedure,  execution,  and  report) 


-  Resources 

-  Duration 


Produce  Artifact 
Share  Data 


Table  references  are  assumed  to  be  developed  correctly. 
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What  We  Covered 


NORTMROP  GRUMMAN 


•  Test  Plan  Values 

•  Test  Plan  Usages 

•  Test  Plan  Needs 

•  Test  Plan  Styles 

•  Design  of  experiments 

•  Test  Plan  Input  Sources 

•  Verification 

•  Resources 


•  How  to  develop  a  test  plan 
model 

•  How  to  optimize  the  plan 

•  How  to  produce  an  artifact 

•  How  to  advance  the  test 
planning  science 
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Conclusions 


NORTHROP  GRUMMAN 


The  DNA  of  T&E  must  Change 


-  Need  a  complete  lifecycle  experience 

•  Test  planning  must  be  recognized  as  the  requirements  set  for  the  test  program 

-  Document  is  not  just  a  deliverable 

-  Plan  does  not  become  extinct 

•  Test  verification  and  planning  techniques 

-  Links  the  systems  engineering  team  with  the  test  team 

-  Enables  collaboration 

-  Fosters  communication 

-  Supports  development  of  early  lifecycle  products 

•  Test  tools  kit  must  be  evolved 

-  Model  based  test  plans  (know  you  have  the  right  plan) 

-  Physics  based  test  event  validation  (know  your  plan  is  right) 

-  Tools  must  be  program  sizable  (big  to  little) 

-  Tools  must  be  connected  to  the  requirements  process 

-  Tools  must  be  collaborative 
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An  Industry  Response  to  the 
Acquisition  Changes 

27th  Annual  National  T&E  Conference 

Marriott  Tampa  Waterside 
March  15th,  2011 

Robert  Sheehan 

Director 

Flight  Test  and  Evaluation  -  Aerospace  Systems 

Steve  Scukanec 

Senior  Test  and  Evaluation  Engineer 
Flight  Test  and  Evaluation  -  Aerospace  Systems 


An  Industry  Response  to  the  Acquisition 
Changes  -  Agenda 


NORTHROP  GRUMMAN 


•  Presentation  Agenda 


-  The  Acquisition  Reform  and  Initiatives 

-  Change  Analysis 

-  T&E  Keys  Acquisition  Tenets 

-  The  Transformation 

-  Conclusions 


Skills 


Processes 


Change 


Tools 


Organization 


“Let  our  advance  worrying  become  advance  thinking  and  planning”  - 

Winston  Churchill 
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An  Industry  Response  to  the  Acquisition 
Changes 


NORTHROP  GRUMMAN 


5000.02  (2008) 

Significantly  Revised  Acquisition  Policy 
Earlier  definition  of  scope,  risk  and  cost 
Mandatory  entry  point 

Special  procedures  for  IT  services  over  $500  million 
Risk  Reduction 

Competitive  prototyping 
Highly  integrated  T&E 

Evolutionary  acquisition  (NOT  spiral  development) 
Enhanced  Oversight 

More/more  frequent  assessments 
Peer  reviews 

Configuration  Steering  Boards 


o&s 


> 


> 


S.454  -  Weapon  Systems  Acquisition  Reform  Act  of  2009 


j.F$roduced 


Senate 

Passed 


O  ,eP,?£id 

“  i  g  n 


dent 

gned 


BILL 

LAW 


02/23/OS  05/O7/OS 


05/22/OS 


Latest  Action  May  22,  2009  Became  Public  Law  No:  111  -23. 


WSARA  (2009)  Kev  Areas  Affecting  T&E 
Creates  Director,  Developmental  Test  &  Evaluation 
•Reviews  and  approves  DT&E  plan  in  the 
TES  and  TEMP  for  MDAPs  and  all  programs 
on  the  OSD  DT&E  Oversight  List 
•Monitors  and  reviews  DT&E  of  MDAPs 
•Has  access  to  all  Component  records  and 
data  necessary  to  carry  out  duties 

DoDI  5000. 02&I  implementation  of  the  Weapon  Systems  Acquisition  Reform  Act  of  2009  &New 
Changes  to  Policy  Karen  Byrd  DAU  Learning  Capabilities  Integration  Center  Learning  Asset 
Program  Manager  May  2010 
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OT&E  Initiatives 


NORTMROP  GRUMMAN 


•Field  new  capability  rapidly 

•Engage  early  to  improve  requirements 

•Integrate  developmental,  live  fire,  and  operational  testing 

•Substantially  improve  suitability  before  Initial  Operational  Test 

&  Evaluation  (IOT&E) 


MEMORANDUM  FOR  DOT&E  STAFF  24-Nov  2009  -  J .  Michael  Gilmore  Director  DOT&E 


Wherever  practicable,  IOT&E  will  be  conducted  using  low-rate  initial  production  (LRIP)  systems  assembled  using  the 
parts,  tools,  and  manufacturing  processes  intended  for  use  in  full-rate  production.  The  system  will  also  utilize  the 
intended  production  versions  of  software.  In  addition,  the  logistics  system  and  maintenance  manuals  intended  for  use  on  the  fielded 
system  should  be  in  place. 

Memo  -  Use  of  Production-Representative  Test  Articles  for  Initial  Operational  Test  and  Evaluation  (IOT&E)  J .  Michael  Gilmore  Director  OT&E  18-0ctober-2010 


-"single  most  important  step.,  .is  to  ...  execute  a  viable  systems  engineering  strategy  from  the  beginning,  including  a  robust 
reliability,  availability,  and  maintainability  (RAM)  program" 

We  know  the  problem  persists.  We  know  that  it  results  in  higher  costs  and  less  effective  systems.  We  know  more  stringent 
engineering  is  required  to  deliver  reliable  products.  To  that  end,  industry  must  be  made  aware  that  all  our  contracts  will 
require,  at  a  minimum,  the  system  engineering  practices  of  ANSI/GEIA  STD-0009. 

MEMORANDUM  FOR  PRINCIPAL  DEPUTY  UNDER  SECRETARY  OF  DEFENSE  (ACQUISITION,  TECHNOLOGY  AND  LOGISTICS)  SUBJECT:  State  of  Reliability  J .  Michael 
Gilmore  Director  OT&E  30-J  une-2010 

T&E  excellence  requires  active  leadership,  sound  planning,  and  realistic  integrated  developmental  testing  (DT)  and 
operational  testing  (OT). 

Incorporating  Test  and  Evaluation  into  Department  of  Defense  Acquisition  Contracts  -  MAY  2009  -  OUSD  ,  AT&L 
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Industry  Analysis  of  Acquisition  Changes 
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Acquisition  Changes 
•  Legislation 
Directives 


Contract 

Funding 

Program  Direction 


Skill  Mix 
Tools 

Organization 

Processes 


•The  New  Policies  Address  The  Hard  Questions  - 
^Provides  Good  Answers 
•Management  of  New  Policies  are  J  ust  Coming  - 

^Program  Management  (Customer  &  Contractor)  Embrace  the  Changes 
•Implementation  Underway - 

^New  Program  Implementation  a  Mixed  Bag,  Change  is  Hard 


Policy  Alone  will  Not  Effect  Change 
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Five  (5)  Acquisition  Keys 

NORTHROP  GRUMMAN 

0f 

Prototyping 

Increase  Modeling  and  Simulation 

Focus  on  Needed  Technology  Development 

Establish  Operational  Environment  Early 

Operational  Realism 

Establish  WIPT  Early 
Early  Test  Planning 
TEMP  Alignment 


Integrated  Testing 

Data  Plans 

Proper  Contract  Language 
Early  Identification  of  Data  Needs 
Evaluate  in  Proper  Environment 


Rapid  Fielding 
Slow  the  Requirements  Growth 
Test  Operationally 
Collaborative  Test  Planning 


RAM 

Early  Manufacturing  Inputs 
Early  RAM  Simulations 
Still  Underwork 


Key  Acquisition  Changes  Which  Drive  T&E  Change 
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The  Transformation 
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Training 
Skill  Mix  Development 
Enhance  Test  Techniques 

Design  Of  Experiments 
Modeling  and  Simulations 
Develop  RAM  Test  SME 


Integrated  Across  Disciplines 

Developed  to  Support  Integrated 

Testing  Concept 

Establish  RAM  T&E  Processes 

Career  Path  Development 

Rotational  Programs  Enhance  Lifecycle 
Experience 


Develop  T&E  Test  Tools 

Test  Plan  Modeling 
Collaborative  Test  Program  Dev. 
Optimized  Test  Program 

Modeling  and  Simulation 

Physic  Based ,  Dynamic 
Test  Event  Evaluations 


Multi-disciplined  Team 

Test  WiPT  Established 
Processes  Based 

Aligned  /  Coordinated  &  Integrated 

Early  Program  Involvement 

Support  Good  Program  Start 
Ensure  T&E  Needs  Identified 
Integrated  Disciplines  From  the  Start 

Customer  Aligned 

Enhances  Communications 


Prototyping 


Integrated 

Testing 


The  T&E  DNA  Requires  Change  to  Implement  the  Policies 
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The  Results 
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XV 


xx 


IVlat'&ri  el 
^  Solution 
^Analysis 


Technology 

Development 


ODD 


Engineering  and 
Man  ufacturing 
Development 


JC  IDS  Process 


PDR 


Post-PDR- 

Assessment 


Production  and 
Deployment 

0 


ODR 


Full  Rate  Production 
Decision  Review 


Phase 

Task 

Value 

Product 

MDD  — 

M/S  A 

•Execute  Architecture  Analysis  * 
•Establish  Long  Lead  Test 

Program  requirements 
•Establish  Integrated  Test  Team 
•Develop  Test  Strategy 
including  Technology 
development  activities 

•Testable  Architecture 

•Initial  Test  Plans  and  Facilities  Definitions  Established. 
•Initial  Requirements  Based  on  Tested  Architecture 
•Architecture  Streamlined  Testable,  Essential 
Requirements  Identified  Aiding  in  Rapid  Deployment 
•Event  Based  Test  Schedule  Developed 

•Integrated  Architecture 

•Schedule 

•Major  Test  Assets 

Identified 

•TES 

TDP 

•Conduct  Requirements 
Verifiability  Assessment  * 

•  Conduct  Verification 
Requirement  Development  * 
•Develop  Test  Unique  Design 
Requirements  * 

•Conduct  Required  Prototyping 
/  Risk  Assessments 
•Establish  Reliability  Program 
•Establish  Test  Program  Plan 

•Verifiable  Requirements  &Verification  Statements 
Development  Avoids  Requirements  rework. 

•Test  Unique  Design  Requirements  Completes  the 
Requirement  Set. 

•Embedded  Operational  Realism  in  Test  Proaram  Helps 
Prove  Product  can  meet  its  intended  use 
•Support  Technology  Assessment  /  Maturation  /  Risk 
Reduction  -  Supply  Valuable  Decision  Data 
•Support  Operational  Sustainment  Assessment 
•Integrated  Test  Program  Developed  and  Coordinated 

•Solid  Requirements 
•Integrated  Test  Program 
Identified  and  Planned 
•Prototyping  data 
•  Initial  RMA  Program 
Established  (ANSI/GEIA 
STD-0009) 

•TEMP 

•Contractor  Test  Plan  Draft 

EMD 

•Requirements  Refined  and 
Allocated 

•Integrated  Test  Planning 
•Facilities  Planning  and 
Development 

•Integrated  Developmental  Test 
Conduct 

•Refined  Verification  Requirements 
•Conduct  Consistent  Test  Program  Through 

Development  Cycle 

•On  Time  Establishment  of  Test  Facilities 
•Coordinated  Contractor  /DT  and  OTTest  Plans 
•Integrated  and  Verified  Product 
•Initial  Operational  Assessments  Supported 

•Traditional  Test  Program 
Executed 

•Product  Verification 
•Integrated  DT  /  OT  Data 
Available 

Production 

•Support  Transition  Support  to 
Manufacturing 

•Integrated  and  Tested  Product 

•Solid  Manufacturing 

Process  based  on  EMD 
Lessons  Learned 

8  *  New  Initiative  To  Improve  Test  Program  Execution 
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Conclusions 


NORTMROP  GRUMMAN 


•  The  Acquisition  Changes  when  implemented  will  make 
effective  changes  to  the  Warfighter  products 

•  The  DNA  of  the  Test  Community  must  change  to 
accommodate  the  intent  of  these  changes 

•  Industry  is  transforming,  policies  and  initiatives  are 
forcing  functions 

•  Early  T&E  can  help  programs  start  right 

•  PMs  must  account  for  early  T&E  in  achieve  the  policy  / 
initiative  intent 


Change  is  slow,  RFP  language  changes  can  increase 
industry  change 
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An  Industry  Response  to  the  Acquisition 
Changes 
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IN  LIFE 

QUESTIONS  ARE  GUARANTEED 

ANSWERS 
NOT  SO  MUCH 

5% 
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An  Industry  Response  to  the  Acquisition 
Changes 
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•  Contact  Information 

-  Robert  Sheehan 

Director  Flight  Test  and  Evaluation 
Northrop  Grumman  Aerospace  Systems 
Robert.Sheehan@NGC.com 
310-332-6927 

-  Steve  Scukanec  “The  Test  Guy” 

Senior  Flight  Test  Engineer 

Northrop  Grumman  Aerospace  Systems 
Stephen.Scukanec@NGC.com 
310-350-3156 
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Marine  Corps  Operational 
Test  &  Evaluation  Activity 


NDIA  27th  Annual  National  Test  and  Evaluation  Conference 
Shannon  Krammes,  Decision  Sciences  Lead 


March  2011 


Impacts  of  the  Learning  Curve 
Operational  Test  and  Evaluation 

Operational  Testing  Challenges 


NDIA  Abstract  (Agenda) 


•  In  the  conduct  of  operational  testing  MCOTEA 
often  experiences  operator  learning  curves. 

•  Operator  learning  curves  can  be  a  nuisance  to 
distinguishing  between  true  system  operational 
performance  and  operator  learning. 

•  How  MCOTEA  assesses  the  learning  curve  prior  to 
commencement  of  the  record  test  portion  of 
operational  testing. 

•  Application  of  the  learning  curve  data  as  a  means 
to  evaluate  new  equipment  training  packages, 
training  systems,  and  formal  training  programs. 
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M COTEA  Mission 


MCOTEA  provides  operational  testing  and 
evaluation  on  behalf  of  the  Marine  Corps 
and  conducts  additional  testing  and 
evaluation  as  required  to  support  the 
Marine  Corps  mission  to  man,  train,  equip 
and  sustain  a  force  in  readiness. 


Why  Do  We  Care? 


•  MCOTEA  evaluates  the  system. ..and  the 

system  includes  the  operators  and  training. 

•  Operational  Test  Readiness  Review 

-  Training  documents  have  been  provided  to  the 
OTA  30  days  prior  to  the  OTRR. 

-Training  has  been  completed  and  is  representative 
of  that  planned  for  fleet  units. 

-The  OT&E  manning  of  the  system  is  adequate 
in. ..experience  level  to  simulate  normal  operating 
conditions. 


What  is  a  Learning  Curve? 


^1 


•  When  several  trials  are  given  in  an  experiment 
and  measures  of  learning  or  of  retention  are 
obtained,  these  measures  may  be  plotted  in 
the  graphic  form  known  as  a  learning  curve,  a 
graph  which  affords  a  comparison  of  the 
performance  on  each  trial  with  a  performance 
on  other  trials.1 
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Learning  Curve  Example 


Dependent  Variable  (reload  time) 
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Other  Lear 


A  Learning  “Curve”  is  far  from  a  straight  progression 


GAMING  SKILL 


Curves 


TJ 

CD 


S' 


3 

0) 

o 

0 


3 

0 

Q) 

0 

C 

0 


Number  of  trials  or  attempts  at  learning 


LEARNING  CURVES  OF  POPULAR  MMORPGS 


■  Wc<1d  of  Warcrafl 

■  POTBS 


□  LOTR  Crfinc 
■  Eve:  The  Second  Genesis 
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Why  Learning  Curves? 


•  Operational  testing  lessons  learned 

-  New  systems 

-  Discovery  of  training  inadequacy 

-  Structure  of  forces  (cohesion) 

•  Distinguishing  between  true  operational 
performance  and  operator  learning 

•  Training  and  proficiency 
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Mitigation  Prior  to  Record  Test 


^1 


•  Test  Design 

-  Randomization 

-  Control 

-  Replication 

•  New  Equipment  Training  evaluation 

•  Extended  pilot  test 

•  Training  certification 

•  Continuous  evaluation 
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Learning  Curve  Applications 


•  New  Equipment  Training  packages 

•  Training  systems 

•  Formal  training  programs 
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Questions? 


Marine  Corps  Operational  Test  and  Evaluation  Activity 

(MCOTEA) 

Shannon  Krammes 

703-432-0945 

shannon.krammes(5)usnnc.mil 


BACKUP  SLIDES 


References 


1.  Garry,  R.,  and  Kingsley,  H.L.  The  Nature  and 
Conditions  of  Learning,  Prentice-Hall,  Inc., 
New  Jersey,  1970. 
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Potential  Models 


•  Henderson's  Law  (power  law  function) 

-  C„  =  Q/r” 

•  Where 

-  CI  is  the  cost  of  the  first  unit  of  production 

-  Cn  is  the  cost  of  the  nth  unit  of  production 

-  n  is  the  cumulative  volume  of  production 

-  a  is  the  elasticity  of  cost  with  regard  to  output 

•  Exponential  Model 


Contact  Information 


•  Shannon  Krammes 

•  703-432-0945 

•  Marine  Corps  Operational  Test  and  Evaluation 
Activity  (M  COTEA) 

•  shannon.krammes@usmc.mil 
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WARFARE  CENTERS 


DAHLGREN 


Low  Cost  UAV  Runways 


Lorenz  Eber,  P.E. 

Unmanned  System  Safety  and  Operations  Director 
NSWC  Dahlgren,  G66  Test  and  Evaluation, 
Telephone:  (540)  653-0728 
lorenz.eber@navv.mil 
March  15,  2011 


DISTRIBUTION  STATEMENT  A:  Distribution  approved  for  public  release; 

distribution  is  unlimited. 


n&sea  Why  UAV  Runways? 

WARFARE  CENTERS  ^ a 


DAHLGREN 


NSWC  Dahlgren  Base 
Runway 


1.  Launch  &  Land  in  Restricted  Airspace  (no  FAA  COA  required) 

2.  Population  or  Building  over-flight  issues 

3.  Separating  Manned  &  Unmanned  Aircraft 

4.  Expeditionary  Runways  for  Theater 

5.  Hazardous  Testing  at  Remote  Sites 
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Runway 
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Future 


UAV 

Runwa' 
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Runwav  Sitin 


Away  from  People  and  Property 
Within  Restricted  Airspace 
Minimal  Terrain 
Minimal  Obstructions 
Minimal  Manned  Air  Traffic 
Align  with  Prevailing  Winds 
Consider  UAV  Traffic  Pattern 
Consider  Environmental  Factors 
Consider  Required  Approvals 


Restricted 

Airspace 
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UAV  Runway  Surface  Types 


Concrete 

Asphalt 

Expeditionary 
Mats  and  Grids 

Dirt 


Chip  Seal 
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Geotextile  Runways 


RC 


=> 

UAV 


•  %  the  cost  of  Asphalt 

•  Can  be  expanded  /  re-configured 

•  Semi-Permanent 

•  3-7  year  life 

•  Permeable  /  Environmentally  Friendly 

•  Can  be  paved  later 
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Runwav  Type  Comparison 


DAHLGREN 


Type 

Approx.  Life 

(Years) 

Approx.  Max 
Wheel  Load 

Approx.  Cost 

(UAV  Application) 

Concrete 

International  Airport 

20-30 

>45,000  lb 

$  38  /SY  (4”) 

Asphalt 

National  Airport 

15-20 

<  35,000  lb 

$  1 8  /SY  (2”) 

GFI  Mats 

Military 

15 

<30,000  lb 

$100/SY 

Dirt 

Private  Airport 

1 

0-30,000lb 

weather  dependent 

$  2.75  /SY 

Chip  Seal 

NZ  Light  Duty  Field 

3-5 

<  5,000lb 

$  6. 25/S Y 
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Geotextile  Runway  Life 


Fabric 


Construction 
can  be 
Phased 


Fabric 
on  Rock 


Asphalt 

over 

Fabric 


•2-3  Year  Life 
•Expeditionary 


•4-7  Year  Life 
•Semi-Permanent 


Asphalt 


■15-20  Year  Life 
■Permanent 


Fabric 


Fabric 
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Cost  Example  1350  ft  x  60  ft 


WARFARE  CENTERS 


•  Clear,  Grub  and  Roll: 


•  3  men,  Dozer,  Loader, 
Truck  and  Roller;  1 
week=  $25,000 

•  Fabric  Cost: 

•  10,350SY  x  $1 .85= 
$19,000 

•  5  men  1  week= 

$22,000 

•  Paint  and  Misc= 

•  $8,000 


•  Clear,  Grub  and  Roll: 
$25,000 

•  Separator  Fabric:  $5,400 
+  $5,300  labor 

•  6”  Crushed  Rock: 
$55,000  +  $25,000  labor 

•  2”  Asphalt: 

$160,000 

•  Paint ,  Drainage  and 
Misc: 

$20,000 
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Planning  CONUS 


Planning  Expeditionary 


•  Site  selection 

•  Approvals 

-  Base  /  Municipality 

-  FAA 

-  Environmental  Permits 

-  Other:  Explosive 

•  Topographic  Survey 

•  Geotechnical  Report 

•  Design 

-  Size 

-  Orientation 

-  Cut  &  Fill 

-  Drainage  (crown  1-2%) 

-  Pavement  Section 

-  Striping 

-  Plans,  Specs  Estimates 

•  Contracts  and  Bids 


Site  selection 
Approvals 

-  Base  /  FOB 

-  Local  Authorities 
Design 

-  Size 

-  Orientation 

-  Striping 

-  Drainage 
Organize  Work  Party 
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Construction  Steps 


DAHLGREN 
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Construction  Eaumment 


Bulldozer 


Surveying  Level 
Grader 


Roller 


Dump  Trucks 
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Construction  Survey 


•  Transfers  the  design  onto  the  ground 

•  Stake  centerline 

•  Elevation  stakes 

•  Survey  Contract  or 
simple  $300  level 
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Excavate  and  Compact 


•  Remove  Organics 

•  Remove 
obstructions 

•  Prepare  and 
compact  sub¬ 
grade 

•  Herbicide  to 
prevent  growth 

•  Pipes  (if  required) 
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Build  up  UAV  Runway  Section 

Expeditionary 


Staked  US  230 
Geotextile  Surface 


Graded  &  Compacted 
Native  soil 
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Build  up  UAV  Runway  Section 
Semi-Permanent 


US  230  Geotextile  Surface 
(not  shown) 

6”  Crushed  Gravel 

US  200  Geotextile 

Fill 

Native  soil 
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Placing  Geotextile  Surface 


DAHLGREN 


•  Check  for  tears  at 
Roll  ends  and 
remove 

•  Keep  Rolls  running 
straight 

•  Apply  Tar  on 
seams  6”  min,  12” 
max  overlap 

•  Anchor  runway 
edges  under  rock  if 
available 
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Use  landscape 
stakes  or  nails  with 
washers  on  1  -2ft 
centers 

Fold  horizontal 
seams  and  nail 

Do  not  pull  fabric  too 
tight.  Leave  some 
minor  wrinkles 

Sun  will  heat  and 
stretch  surface  ‘drum 
tight’ 


8”  Nail 
with 

4  washer 


/  wv !%  v '  J 
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•  Use  Temporary  X’s 

•  Follow  FAA  Standards  for 
Airport  Markings 

AC  NO.  150/5340-1 

•  Do  Not  use  Runway 
number  markings.  Use 
‘UAV’  instead 

•  Use  large  60’  x  60’  Yellow 
Xs  every  1 000’  per  AC 
150/5340-1 

•  Standard  Latex  Road 
paint 

•  Paint  ‘Rotor  Wing 
Prohibited’  in  20’  letters 
on  center  of  runway 
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Final  Touches 
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•  Prevent  Vehicles 
from  driving  on 
runway 

•  Remove  Flight 
Obstructions 

•  Place  Wind  Sock 

•  Tar  over  Nails 

•  Seeding 

•  Access  Ramps 

•  Electrical  hook-up 
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Upkeep 


DAHLGREN 


•  Walk  Runway  before 
every  flight 

•  Remove  debris  and 
weeds 


•  Sweep  if  required 

•  Repair  rips  and  tears 
with  tar  and  patches 


•  Check  for  protruding 
Nails  /  Stakes 
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Questions  ? 


imAvsea 
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Marine  Corps  Operational 
Test  &  Evaluation  Activity 


NDIA  27th  Annual  National  Test  and  Evaluation  Conference 

LtCol  Michael  Kennedy 


March  2011 


Effective  Combat  Data  Collection  & 

Applicability  to  T&E 


Agenda 


•  Background 

•  Command  Relationships 

•  Applicability 

•  Collection  Methods 

•  Data  Opportunities 

•  Limitations 

•  Evaluation 

•  FOAXV 

•  Questions 


Background 


Forward  Operational  Assessment  Data  Collection: 

2003  -  Department  of  the  Army  G3/G8  tasking  designated  ATEC  as  the  primary 
agent  for  OIF/OEF  system  assessments 

2003  -  Operational  Assessment  (OA)  team  I  deploys  for  47  days  to  Afghanistan, 

Iraq,  and  Kuwait 

2004  -  OA  team  II  deploys  for  33  days  to  Afghanistan  and  Kuwait 

2005  -  OA  team  III  begins  the  first  six  month  rotation  of  a  continuing  FOA 

presence  to  conduct  operational  assessments  on  systems  that  included 
Rapid  fielding  initiatives  (RFIs)  and  equipment  procured  due  to  urgent 
materiel  release  (UMR)  requests  generated  in  theater 

2010  -  Discussions  between  MCOTEA  and  ATEC  resulted  in  the  attachment  of  a 

MCOTEA  Assessment  team  with  FOA  XV  from  August  2010  through 
February  2011 

2011  -  FOA  XVI  is  currently  deployed  with  a  MCOTEA  team  attached 


Command  Relationships 


•  Component  Command 

•  ISAF  Regional  Commands 

•  Parent  Command 

•  Mobility /Transportation 

•  Logistics  and  Communications 


Applicability 


•  Data  collected  in  a  forward  operations  area 
can  be  used  to  supplement  current,  validate 
previous  and  guide  future  test  and  evaluation 

•  Data  collected  is  from  the  system  as  used  by 
deployed  forces  conducting  actual  missions 
and  is  subject  to  variables  that  would 
confound  formal  T&E 

•  Assessments  based  on  available  data,  not 
controlled  test  events 


Collection  Methods 


•  Clearly  identify  your  Data  Collection  Plan 

•  Understand  the  reality  of  the  environment 

•  Flexibility  during  execution 

•  Minimize  impact  on  Operational  Units 

•  One  chance  to  get  it  right 

•  CONUS  data  collection  is  preferred 


Data  Opportunities 


•  Maintenance  Reports 

•  Logistics  Reports 

•  Operations  Logs  /  SIGEVENTS 

•  Electronic  Data  Collection 

•  Integrated  Data  Collection 

•  Forensics 

•  After  Action  Reports 

•  Surveys 

•  Interactive 


Limitations 


•  Mission  /  Current  Operations 

•  Access  to  system  under  assessment 

•  Availability  of  test  instrumentation 

•  Unit  exposure  to  system 

•  Tactics,  Techniques  and  Procedures 

•  Environment 

•  Unit  personnel  turnover 


Evaluation 


•  Understand  what  information  decision  makers  need  / 
want 

•  Assessment  Evaluations  should  provide  Operational  Force 
Commanders  with  timely,  concise,  understandable 
information 

•  FOA  is  only  one  of  multiple  sources  for  gathering 
information 

•  Performance  Conclusions  should  be  limited  to  Data 
Adequacy  inherent  in  a  combat  environment 

•  Operational  Assessment  data  evaluation  can  identify 
issues  for  further  investigation  through  formal  T&E 


FOA  XV 


•  Systems  assessed  for  USMC: 

•  Counter  I  ED  Systems 

•  Experimental  (Green)  Energy  Systems 


Biometrics  Systems 


WOtt* 
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Questions 


LtCol  Michael  Kennedy 
Marine  Corps  Operational  Test  and  Evaluation  Ac 
michael.kennedy  @  usmc.mil 
703-432-8059 


ivity  (MCOTEA) 


OSD 
in  Navy 

Do  Additional  DT&E  Opportunities  Exist? 


Perspective  of  DT&E 
Shipbuilding  Programs 


Mr.  Patrick  Clancy 
Deputy  Director,  Naval  Warfare 


Mr.  Joe  Terlizzese,  NW  AO 
Mr.  Michael  Melvin,  NW  AO 
ODASD(DT&E) 
703-697-5733 
Patrick.Clancy@osd.mil 


15  March  2011 


Outline 


Shipbuilding  vs.  other  DoD  acquisitions 
Challenge  of  Shipbuilding  DT&E 
New  Approach  for  DT&E  on  Ships 
Shipbuilding  DT&E  Best  Practices 
PARMs 
Summary 
Q  &  A 
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Shipbuilding  vs.  Other  DoD  Acquisitions 


•  Limited  use  of  Prototypes,  EDMs,  “Fly-before  buy” 

-  Prohibitive  cost  for  test  articles 

•  Larger  Scope 

-  Long  construction  time  leads  to  parallel  design  and  building 

•  Complexity 

-  Many  programs  in  one  (i.e.,  weapons,  propulsion, 
aviation,  C4I,  navigation,  habitability,  etc.) 

•  System-of-systems  (SoS) 

-  Virtually  all  mission  capabilities  require  interaction  of  numerous  sub-systems 
and  components 

-  Many  SoS  consist  of  mix  of  new  and  old  systems  or  components 

•  Performance  and  schedule  highly  dependent  on  Participating 
Acquisition  Resource  Managers  (PARMs) 

(  ~  'N 

Shipbuilding  T&E  Process 
Inherently  Leads  to  a  Different  T&E  Approach 
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Challenge  of  Shipbuilding  DT&E 


•  First  ship  is  the  test  article  in  shipbuilding  T&E 

-  Is  ultimately  a  production  article 

-  Often  no  time  for  test-analyze-fix  in  shipbuilding  trials 

-  Multiple  follow-on  ships  being  built  while  DT/OT  being 
conducted  on  first  of  class 


•  Fixes  often  limited  to  mission-critical  discrepancies 

•  Lower  priority  discrepancies  are  often  forward  fit  to  future  hulls 

-  Possible  back-fit  to  early  hull(s)  during  future  maintenance  cycle 

C  \ 


Shinbuildina  DT&F 
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A  New  Approach  for  DT&E  on  Ships 


•  Opportunities  for  concurrent  DT&E  and  OT&E 
Shipbuilding  T&E  continuum 

-  Industrial  Stage  Tests 

-  Fast  Cruise 

-  Builder’s  Trials 

-  Acceptance  Trials 

-  Post  Delivery  T est  and  T rials 

-  Certifications 

•  Aviation,  ATO,  HERO,  UNREP  SQT,  CSSQT,etc 

•  Eliminate  duplication,  optimize  efficiencies,  increase  opportunities  to 
find  &  fix  problems 

•  Requires  access,  partnerships,  data  sharing  --  represents 
challenges 

•  A  true  acceptance  of  Integrated  Testing  across  the  T&E  continuum 


throughout 


Taking  Credit  for  ALL  TESTING 


j 
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Shipbuilding  DT&E  Best 


r 


v 


Critical  Risk  Mitigation  is  done  on  Major 
Components  at  Land-Based  Test  Sites 

-  Surface  Combat  Systems  Center,  Wallops  Is 

•  SSDS,  AEGIS,  DDG  1000 

-  Test  &  Integration  Facility  (TIF),  Charleston,  SC 

-  NAVSEA  Panama  City  -  LCS  MCM  MP 

-  NAVSEA  Dahlgren  -  LCS  SUW  MP 

-  NUWC,  Newport,  Rl  -  LCS  ASW  MP 

-  DDG  1000  Integrated  Power  System  LBTS, 
Philadelphia,  PA 

-  NAVAIR,  EMALS/AAG,  Lakehurst,  NJ 

-  NAVSEA  Carderock,  Acoustic  Research 
Detachment  -  Lake  Pend  Oreille,  Idaho 

-  VASCIC,  CVN-78, Newport  News,  VA 

-  COATS,  SSN-774, Groton,  CT 


What  Other  Testing  is  Being  Done 
That  Can  be  Used  for  DT&E  Credit  to 
Reduce  Risk  going  into  OT? 
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PARMs 


•  Participating  Acquisition  Resource  Managers  (PARMs)  are 
responsible  for  developing  their  system  independently,  while 
meeting  a  defined  in-yard  date 

-  Usually  not  under  shipbuilding  PM  control 

•  Relieves  workload/But  no  direct  authority 

-  PARM  can  be  resident  from  different  PEO  or  SYSCOM 

-  Matrix  like:  PM  funds  task/PARM  funds  staff 

•  PARMs  add  flexibility  and  efficiency  by  developing  systems  and 
equipment  in  parallel  with  ship  construction 

-  Ship  PM  defines  interface  specs 

-  PARM  develops  sub-system  solution 

-  Ship  schedule,  cost  and  performance  highly  dependent  on  PARMs 

•  Challenge:  Who  is  the  systems  integrator? 


(  ^ 

PARMs  -  Big  Payoff  if  Successful 

_ 
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Summary 


•  Shipbuilding  is  different  from  other  acquisition  programs 

-  Our  approach  to  Shipbuilding  T&E  also  needs  to  be  different 

-  Shipbuilding  has  a  long  cycle  time  to  complete  a  test  article 

-  Test  article  is  always  a  production  article 

-  Multiple  follow-on  ships  are  already  well  into  construction  when  DT/OT  are 
being  conducted 

-  All  “fixes”  need  to  be  incorporated  on  all  of  these  ships  post-test 

•  Ships  and  their  major  components  go  through  a  plethora  of  testing 
before  DT/OT 

-  Many  of  these  can  be  used  as  opportunities  for  DT/OT 

-  Use  of  LBTS  is  a  best  practice  that  pays  dividends 

-  What  other  testing  is  being  done  that  can  contribute  to  DT&E? 

•  Must  take  advantage  and  credit  for  developmental  testing 

-  Will  ultimately  lead  to  more  efficient  and  successful  development 
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Points  of  Contact 


Patrick  Clancy 

Deputy  Director,  Naval  Warfare 
ODASD(DT&E) 
703-697-5733 
Patrick.Clancy@osd.mil 


Joe  Terlizzese 

Action  Officer 
703-412-3687 

Joseph.terlizzese.ctr@osd.mil 


Michael  Melvin 

Action  Officer 
703-412-3661 

Michael.melvin.ctr@osd.mil 


f  \ 

Visit  our  website:  http://www.acq.osd.mil/dte/ 

\ _ _ _ _ _ ) 


Back-ups 
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Does  NAVSEA  Have  an  RTO? 


•  Not  by  name,  but  many  programs  have  an  RTO  by  function 

•  Example:  NAVSEA  Port  Hueneme  Division  (NAVSEA  PHD)  is  non- 
AEGIS  ship  combat  system  RTO 

-  SSDS  In-Service  Engineering  Agent  (ISEA) 

-  Combat  systems  test  lead  for  CVN,  LHA,  LHD,  LPD,  LSD  ship  classes 

-  Operates  the  Self  Defense  Test  Ship 

-  With  NAVSEA  Dahlgren  Division,  performs  systems  integration  at  the 
Carrier  and  Amphib  Land  Based  Test  Site  at  Wallops  Island,  VA 

-  Test  conductor  for  all  DT&E  events  on  Pt.  Mugu,  CA  range 

-  Frequently  assigned  as  COMOPTEVFOR  trusted  agent  for  OT&E  data 
collection  and  test  support 


11 


Navy  Gate  Review  Process 
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Resources. 
Responsiveness, 
liability 


Large  company  practices.  Small  company  responsiveness.  Working  for  YOU. 


Britt  Bray 

Department  Manager  and  Senior  Military  Analyst 

15  March  2011 


■ 


Purpose  and  Agenda 


M  Purpose:  To  explain  the  process  and  logic  for  specifying  an 
understanding  of  the  mission  (MBTE  Step  1) 

►I  Agenda 

►  Introduction 

►  Where  the  Mission  fits  in  the  MBT&E  framework 

►  Warfighting  101  -  Analyzing  the  Mission 

>  Mission?  What  Mission?? 

>  Analyzing  the  mission 

>  Decomposing  the  mission  into  tasks 

>  Determining  conditions  and  standards  (i.e.  MoPs  and  MoEs ) 

►  Translating  tasks  from  native  language  to  common  language 

>  Authoritative  Task  Lists 

>  Capturing  the  results 

>  An  alternative  approach 

►  Questions  to  ask 
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Mission  Role  in  MBTE 


Why?  -  Acquisition  Initiatives 

Common  Focus  on  Mission  Capability 

- DoD - 

POD  5000.1  -  “The  primary  objective  of  Defense  acquisition  is  to  acquire  quality 
products  that  satisfy  user  needs  with  measurable  improvements  to  mission 
capability  T1 

- JCS - 

Joint  Capabilities  Integration  and  Development  System  -  The  primary 
objective  of  the  JCIDS  process  is  to  ensure  the  capabilities  required  by 
the  joint  yyarjahter  are  identified  ...  in  order  to  successfully  execute 
the  missions  assigned."2 

-  DOT&E  - 

Director.  Operational  Test  and  Evaluation  -  "The  evaluation  of 

operational  effectiveness  is  linked  to  mission  accomplishment.”3 


Goal:  T&E  Focused  on  Mission  Capability 


Army  Proven 

Rntilf*  Bf*nr1v 


■  Oic«  of  m«  unoer  Secret**  e*Oe<erse  Acausieer  -ecrraefiy  e.-a  LogHBes 
:  oace  of  me  Joint  ewe*  cummer  cf  me  J«rt  erne's  c' Sts'*  rewicoec 

;  2zZ  :c~s=  2:*  :•!  -«r:l 


s'  ee^e-se  ;  fectiv e  Nu-noef  sccc  ;  •  ’ey  eccs 
i  MerCS 
6  Jen  10 


What?  -  Framework  Building  Block 

Capability1  -  The  ability  to  achieve  a  desired  effect  [or  result, 
outcome,  or  consequence  of  a  task2] . . . 

-  underspecified  standards  and  conditions 

-  through  a  combination  of  means  and  ways 

-  to  perform  a  set  of  tasks. 


Higher  Level  Task/Action  or 
Desired  End  State 


Means 

Organization  (forces,  units).  Training. 
Materiel  (equipment  functions  & 
resources).  Personnel  and  Facilities. 


Ways 

Doctrine  (tactics,  techniques  and 
procedures).  Leadership  and 
Education,  concepts  and  policies. 


Army  Proven 

Rnfilt a  P pnrlv 


1.  CJCSI 31 70.01  F.  May  2007 

2.  Taken  from  JP 1-02.  Mar  2007.  definition  of  effect. 
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MBT&E  Framework  -  Where’s  the  Mission? 


MISSION  AND  SYSTEM 


EVALUATED  BY 


TESTED  BY 


V 

/ 


4 


■ 


Mission?  What  Mission?? 


M  For  MBT&E  purposes,  want  to  know  “What”  the  unit  employing 
the  system  under  test  is  supposed  to  do,  and  “Why”. 

►  In  the  context  of  at  least  the  next  higher  level  headquarters  mission 

►  And  a  broader  operational  context  (i.e.  Operational  Environment  (OE)  and 
Concept  of  Operations  (CONOPS)) 

►I  What  are  some  potential  sources  for  the  Mission? 

►  Requirements  documents  from  JCIDS/CBA  analysis 

►  Army  Functional  Concept  (AFC)  or  Concept  Capability  Document  (CCD) 

►  CONOPS  based  on  ongoing  operations 

►  Approved  Defense  Planning  Scenarios  (DPS)  or  Army  Scenarios  based 
on  DPS  -  Often  used  for  Analysis  of  Alternatives  (AoA) 

►  Lower  level,  higher  fidelity  vignettes  based  on  above 

►  Other  CONOPS  directed  or  approved  for  use  by  appropriate  authority  (i.e. 
Test  Director,  MDA,  etc.) 
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Analyzing  the  Mission 


►I  FM  5.0,  The  Operations  Process,  dated  March  2010,  describes  the 
Military  Decision  Making  Process  (MDMP). 

►  Mission  Analysis  is  step  2  of  the  MDMP 


Figure  B-1 .  The  steps  of  the  mil  itary  deaHOfmating  process 


Figure  B-2.  Mission  analysis 
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Decomposing  the  Mission  into  Tasks 


►I  Determine  Specified,  Implied  and  Essential  Tasks 

-  The  “what”  of  a  mission  statement  is  always  a  task. 

-  Analysis  of  the  higher  headquarters’  order  and  commander’s  guidance 
identifies  specified  and  implied  tasks. 

-  In  the  context  of  operations,  a  task  is  clearly  defined  as  a  measurable  activity 
accomplished  by  Soldiers,  units,  and  organizations  that  may  support  or  be 
supported  by  other  tasks. 

-  Essential  tasks  are  derived  from  the  list  of  specified  and  implied  tasks,  and 
incorporated  in  the  restated  mission. 

A  specified  task  is  a  task  specifically  assigned  to  a  unit  by  its  higher 
headquarters 

An  implied  task  is  a  task  that  must  be  performed  to  accomplish  a 
specified  task  or  mission  but  is  not  stated  in  the  higher  headquarters’ 
order 

An  essential  task  is  a  specified  or  implied  task  that  must  be  executed  to 
accomplish  the  mission.  Essential  tasks  are  always  included  in  the 
unit’s  mission  statement 
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■ 


Determining  Conditions  and  Standards 


►I  Conditions 

►  Definition:  (joint)  Those  variables  of  an  operational  environment  or  situation 
in  which  a  unit,  system,  or  individual  is  expected  to  operate  and  may  affect 
performance.  (JP 1-02) 

►  Condition  variables  that  may  effect  performance  are  typically  identified 
during  the  Intelligence  Preparation  of  the  Battlefield  (IPB)  process  and  further 
refined  as  a  result  of  Course  of  Action  (COA)  Analysis,  or  Wargaming. 

►I  Standards 

►  Definition:  A  quantitative  or  qualitative  measure  and  criterion  for  specifying 
the  levels  of  performance  of  a  task.(FM  7-0) 

►  For  Mission  Based  Testing  and  Evaluation  we  want  to  determine  Measures  of 
Effectiveness  (MoE)  to  measure  whether  a  task  had  or  is  having  the  desired 
effect;  and,  Measures  of  Performance  (MoP)  to  determine  whether  task 
performance  meets  or  exceeds  the  minimum  criterion  for  performance. 

►  Standards  are  derived  from  analysis  of  what  it  will  take  to  accomplish 
assigned  mission  and  tasks  -  normally  via  the  wargaming  process. 
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Translating  Tasks  into  ATL  Tasks 


►I  Sample  task  list  from  mission 
analysis  process 


►I  Corresponding  tasks  drawn  from 
AUTL  and  other  ATLs 


► 


► 


Specified  Tasks 

>  Move  along  Route  Charlie 

>  Occupy  Assembly  Area  (AA)  Mike 

>  Secure  roads  and  bridges  leading  into 
and  out  of  Town  in  order  to  prevent 
infiltration  by  insurgents 

Implied  Tasks 

>  Maintain  situational  awarenes 

>  Recon  Route  Charlie 

>  Detect,  locate  and  clear  lEDs 

>  Recon  AA  Mike 


Maintain  perimeter  security 
Establish  traffic  control  points 


->  ► 

► 


THE  ARMY 
UNIVERSAL  TASK 
LIST 


ART  1.3.3  Conduct  Tactical  Convoy 
ART  1.5.1  Occupy  an  Assembly  Area 
ART  7.5.19  Isolate  an  Enemy  Force 

ART  6.4.2  Maintain  constant  situational 
awareness 

ART  2.3.3.1  Conduct  a  Route  Recon 

ART  1.6.1  Overcome  barriers, 
obstacles,  and  mines 
>  ART  6.12.3  Conduct  IED  Operations 
ART  2.3.3  Conduct  an  Area  Recon 

ART  6.5.3.3  Establish  Perimeter 
Security 

ART  6.5.3.2  Establish  Checkpoints 
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■ 


Capturing  the  Results 


►I  How  is  it  done  now? 

►  Manual  look  up? 

►  Cut  and  paste  from  requirement  documents? 

►  Captured  in  PowerPoint,  Word  Tables,  Excel  Spreadsheets? 

►I  Why  this  may  not  be  sufficient 

►  Manual  look  up 

>  Currency  of  authoritative  sources  -  ATL’s  updated  regularly 

>  Time  consuming  process 

>  Increased  chance  of  omission  and/or  fat-finger  errors 

►  Cut  and  paste  from  requirement  documents 

>  Conditions  may  have  changed  since  publication 

>  Potential  for  adopting  faulty  or  incomplete  analysis 

►  Captured  in  PowerPoint,  Word  Tables,  etc. 

>  Suitability  for  Re-Use  and/or  collaborative  development 


Page  10 
3/17/2011 


An  Alternative  Approach 


►I  Online,  collaborative  knowledge  capture  tool 

►  Use  to  support  the  following  functions: 

>  Determine  and  document  mission  requirements  in  the  form  of  tasks, 
conditions  and  standards  for  systems  under  test  and  associated  operational 
context 

>  Develop  and  document  planning  for  Test  and  Evaluation  events 

>  Record  and  maintain  task  execution  results 

>  Record  and  maintain  resulting  assessment  for  each  task 

►  Participation  and  permissions  limited  to  members  of  authorized  user 
groups 
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How  it  Works 


I  Determine  and  document  mission  requirements  -  Converting  to  ATL  Tasks 


] India  sample  missionI 


f  -  [3  ART  1 .3.3  CONDUCT  TACTICAL  CONVOY 
C3  Staff  Tasks 

f  C3  ART  6.4.2  MAINTAIN  CONSTANT  SITUATIONAL  AWARENESS 
f  StaffTasks 

^  C3  ART  5.1 .4.1  MONITOR  SITUATION  OR  PROGRESS  OF  OPERAT 
n  ART  5. 3. 5. 2.1  Collect  Friendly  Force  Information  Requirements 
*-  ART  5. 3. 5. 2. 2  Integrate  Intelligence  Products 

C3  ART  5. 3. 5. 2. 4  Process  Relevant  Information  to  Create  a  Commo 
n  ART  5. 3. 5. 2. 5  Display  a  Common  Operational  Picture  Tailored  tc 
O"  C3  ART  5. 3. 5. 2. 6  Store  Relevant  Information 
0  Subordinate  UnitTasks 
Q  Command-Linked  Tasks 
?  C3  Subordinate  UnitTasks 

F  [3  art  2.3.3.1  CONDUCT  A  ROUTE  RECONNAISSANCE 
»  [3  ART  2.3.4  CONDUCT  SURVEILLANCE 
o-  [3  ART  6  5.3.4  ESTABLISH  OBSERVATION  POSTS 
0  Command-Linked  Tasks 
^-[3  ART  1.5.1  OCCUPY  AN  ASSEMBLY  AREA 
Q  StaffTasks 

7  l~3  Subordinate  UnitTasks 

o-  [3  ART  2  3.3.3  CONDUCT  AN  AREA  RECONNAISSANCE 
Q  Command-Linked  Tasks 
[3  ART  6. 5. 3. 3  ESTABLISH  PERIMETER  SECURITY 
0  StaffTasks 

o-  [3  Subordinate  UnitTasks 
0  Command-Linked  Tasks 
9-  [3  ART  7.5.19  ISOLATE  AN  ENEMY  FORCE 
StaffTasks 

V  l~1  Subordinate  UnitTasks 

^  [3  ART  2.3.4  CONDUCT  SURVEILLANCE 
C3  ART  6. 5. 3.2  ESTABLISH  CHECKPOINTS 
o-  ART  6. 5. 3.4  ESTABLISH  OBSERVATION  POSTS 

0  Command-Linked  Tasks 


Mission  Conditions  ['  Doctrine  *  Guidance  Notes  [  MTfls 


Security  Classification: 
Name: 

Code: 

Combatant  Command: 
Last  Modified: 

Created: 


(U) 


NDIA  SAMPLE  MISSION 


US  Army-201 1-0001 
IBCT  INF  BN -RIFLE  CO 
2011-03-04  11:40:26.0 
2011-03-04  10:41:38.0 
Subordinate  Command:  IBCT  INF  BN  -  RIFLE  CO/RIFLE  PLT/RIFLE  SQD 


Modify... 


Modify... 


*  OPLAN: 

Published: 

Level: 


Modify.. 


U 


■  Primary  Mission  (M) 
O  Operation  (O) 

O  Phase (P) 

O  Specified  Task  (S) 
O  Implied  Task  (I) 


Zoom... 


*  Description: 

Company  has  been  assigned  the  mission  of  securing  the  roads  and  bridges  in  and  out  of  a  small  town  in 
their  Area  of  Operations  in  orderto  prevent  insurgents  from  infiltrating  or  exflltrating. 
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How  it  Works  cont. 


►i 


Determine  and  document  mission  requirements  -  Conditions 

f  Mission  Task  Conditions  '  Standards  |  Notes  Observations 


[3  NDIA  sample  mission 
f  C3  ART  1.3.3  CONDUCT  TACTICAL  CONVOY 
9  a  Staff  Tasks 

?  -|l3  ART  6.4.2  MAINTAIN  CONSTANT  SITUATIONAL  AWARENESS 
|  Il5  Staff  Tasks 

^  ART  5.1 .4.1  MONITOR  SITUATION  OR  PROGRESS  OF  OPERATIONS 
*-  ART  5. 3. 5. 2.1  Collect  Friendly  Force  Information  Requirements 
«“  ART  5. 3. 5. 2. 2  Integrate  Intelligence  Products 

l~T  ART  5. 3. 5. 2. 4  Process  Relevant  Information  to  Create  a  Common  Operal 
o-  [3  ART  5. 3. 5. 2. 5  Display  a  Common  Operational  Picture  Tailored  to  User  Ni 
o-  [3  ART  5. 3. 5. 2. 6  Store  Relevant  Information 
0  Subordinate  UnitTasks 
Q  Command-Linked  Tasks 
^  l~1  Subordinate  UnitTasks 

ART  2. 3. 3.1  CONDUCT  A  ROUTE  RECONNAISSANCE 
Q  Staff  Tasks 

f  \~1  Subordinate  UnitTasks 

o-  Q  ART  1 .6.1  OVERCOME  BARRIERS,  OBSTACLES,  AND  MINES 

n  Command-Linked  Tasks 
[3  ART  2.3.4  CONDUCT  SURVEILLANCE 
[3  ART  6. 5. 3. 4  ESTABLISH  OBSERVATION  POSTS 
Q  Command-Linked  Tasks 
?  ART  1 .5.1  OCCUPY  AN  ASSEMBLY  AREA 
[j  Staff  Tasks 
l~1  Subordinate  UnitTasks 

o-  [3  ART  2. 3. 3. 3  CONDUCT  AN  AREA  RECONNAISSANCE 
D  Command-Linked  Tasks 
t  C3  ART  6. 5. 3. 3  ESTABLISH  PERIMETER  SECURITY 
Q  Staff  Tasks 

o-  C3  Subordinate  UnitTasks 
Q  Command-Linked  Tasks 
V  C3  ART  7.5.1  9  ISOLATE  AN  ENEMY  FORCE 
StaffTasks 

$  n  Subordinate  UnitTasks 

ri-  r~^  j 


n  Designators 
Y  General 

^  rlci.1 .3.4  Obstacles  to  Movement 

Q  Moderate  (some  use  of  obstacles) 
n  C  1 .1 .3  Man-Made  Terrain  Features 

Q  Moderate  (impact  on  specific  small  areas) 

C  1 .3.1 .1  Season 
O  Summer  (hot  long  days) 

C  1 .3.1  Climate 
0  Temperate 
C  1 .3.5  RF  Spectrum 
n  Moderate  (some  limiting  factors) 

C  2.4.2  Intelligence  Data  Base 

0  Marginal  (intelligence  data  is  neither  current  nor  complete) 
C  2.9.2  Threat  Form 
0  Unconventional  (guerrilla  warfare) 


t  a 

Cl 

D 

?  a 

Cl 

D 

<p-a 

Cl 

D 

?  □ 

Cl 

D 

fa 

Cl 

D 

?  a 

C  2. 

D 

?  a 

C  2 

D 

Change  Class... 


Add... 

- rage 
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How  it  Works  cont 


►I  Determine  and  document  mission  requirements  -  Standards 


d  ndia  sample  mission 

4  d  ART  1.3. 3  CONDUCT  TACTICAL  CONVOY 
i~d  StaffTasks 

9  d  ART  6.4.2  MAINTAIN  CONSTANT  SITUATIONAL  AWARENESS 
'►-[3  Staff  Tasks 

j>-  d  ART  5.1 .4.1  MONITOR  SITUATION  OR  PROGRESS  OF  OPERATIONS 
*-  d  ART  5. 3. 5. 2.1  Collect  Friendly  Force  Information  Requirements 
d  ART  5. 3. 5. 2. 2  Integrate  Intelligence  Products 
*-  d  ART  5. 3. 5. 2. 4  Process  Relevant  Information  to  Create  a  Common  Operat 
o-  d  ART  5. 3. 5. 2. 5  Display  a  Common  Operational  Picture  Tailored  to  User  Ni 
o-  d  ART  5. 3. 5. 2. 6  Store  Relevant  Information 
0  Subordinate  UnitTasks 
0  Command-Linked  Tasks 
fd  Subordinate  UnitTasks 

^  [3  ART  2. 3. 3.1  CONDUCT  A  ROUTE  RECONNAISSANCE 
Q  StaffTasks 

y-d  Subordinate  UnitTasks 

o-  d  ART  1 .6.1  OVERCOME  BARRIERS,  OBSTACLES,  AND  MINES 

o-  [3  Command-Linked  Tasks 
o-  [3  ART  2.3.4  CONDUCT  SURVEILLANCE 
o-  [3  ART  6.5.3  4  ESTABLISH  OBSERVATION  POSTS 
0  Command-Linked  Tasks 
7  [3  ART  1 .5.1  OCCUPY  AN  ASSEMBLY  AREA 
Q  StaffTasks 

Subordinate  UnitTasks 

O-  [3  ART  2.3.3  3  CONDUCT  AN  AREA  RECONNAISSANCE 
Q  Command-Linked  Tasks 
'f  □  ART 6. 5.3.3  ESTABLISH  PERIMETER  SECURITY 
Q  StaffTasks 

«■  C3  Subordinate  UnitTasks 
0  Command-Linked  Tasks 


1 H  [3  ART  7.5.1 9  ISOLATE  AN  ENEMY  FORCE 
4-  [3  StaffTasks 
ri  Subordinate  UnitTasks 


ADT  -3  -3  A  CCi  Mm  \r-T  Ol  IDWCII 


am  r-n. 


Mission  Task  [  Conditions  Standards  Notes  [  Observations  [  Doctrine  *  [  TPAs  * 


M# 

Sec.  Class. 

Criterion 

Scale 

Measure 

1 

<U) 

20  min 

Time 

(U)  To  conduct  reconnaissance  of  obstacle 

2 

<U) 

60  min 

Time 

(U)  To  complete  obstacle  clearing 

3 

<U! 

0 

Number 

(U)  Of  friendly/neutral  casualties  caused  by  mine/IED  det... 

Add...  1 1  Details  1 1  Remove... 
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Mission  Task  Report  in  TEO  Format 


NDIA  SAMPLE  [Compatibility  Mode]  -  Microsoft  Word 


t  Page  Layout 

Ref  e  re  n  ce  s  Mailings 

Review  View  Add-Ins 

Times  New  Roman 

-  io  -  | A"  at|^I 

||5=  - 1=  -  wll^^ifnlNli 

AaBbCcDc 

AaBbCcDc  AaBbCcDc 

AaBbt 

AaBbCcE 

AoBbCcDt 

:  4k 

r 

B  I  U  at* 

x,  x'  Aa-|  .A  ~ 

11  Normal 

H  No  Spaci...  Heading  6 

Title 

Subtitle 

Subtle  Em... 

_  Change 

T  Styles  T 

i 

Font  ^  | 

Paragraph  [*J 

Styles 

m 

...  g  , 

.  .  !■■■!■■■ 

7  ...  |  ...  2  ...  |  . 

.  .  4  ,  . 

.  |  ...  5  ...  | 

■  ■  ■  6  ■ 

.  7  ,  . 

Mission:  (U)KDIA  SAMPLE  MISSION 

Task:  (U)ART  1.3.3  CONDUCT  TACTICAL  CONVOY 

Responsible  Organization:  IBCT  INF  BN  -  RIFLE  CO 

Description:  1-52.  Conduct  tactical  convoys  by  employing  one  or  a  combination  of  three  types  of  column  formations:  openr  closer 
and  infiltr  ation.  Tactical  convoys  are  combat  operations  in  which  forces  and  materiel  are  moved  overland  from  one  location  on  the 
battlefield  to  another  while  maintaining  the  ability  to  aggressively  respond  to  enemy  attempts  to  impede,  dismpt.  or  destroy  elements 
of  the  convoy.  (FM  55-30)  (ALMC) 

Standards: 


M# 

Criterion 

Scale 

Measure 

Ml 

CP) 

Y/N 

(U)  Hie  SIR  that  prompted  the  conduct  of 
re  c  onnai  ss  anc  e  wa s  an  swered. 

M2 

<P) 

YN 

(U)  Reconnaissance  system  force  orients  on  the 
rec  onnai ssanc  e  objective. 

M3 

<u) 

YN 

(U)  Rec  on  system  force  reports  all  information 
rapidly  and  accurately. 

M4 

CP) 

YN 

(U)  Re  con  mission  completed  no  later  than  time 
specified  in  the  order. 

M5 

CP) 

YN 

(U)  Support  requir  ements  for  each  reconnaissance 
asset  are  identified. 

M6 

CP) 

Y/N 

(U)  Unit  maintains  continuous  reconnaissance  by 
emp  1  o  v  ing  mu  It  ipl  e  me  ans. 

M7 

<P) 

Time 

(U)  From  receipt  of  tasking  until  reconnaissance 
assets  are  in  place. 

MS 

CP) 

Time 

(U)  To  provide  answers  to  IR  PIRto  requesting 
agency. 

DRcS 
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How  it  Works  cont 


►I  Develop  and  document  planning  for  test  events 


Event:  Test  Event  1 


l~^|Test  Event  1 

n  Training  Audiences 
n  Forces 

Q  Linked  Participants 


Locations  Plans *  Costs  *  Transportation  * 

Shortfalls  *  More  Event  Data  * 

Equipment 1  Sim  Support  * 

POCs 

Remarks  Facilities  * 

Component  Interoperability  * 

Event 

Purpose  Description  Methods  * 

Milestones  Goals  * 

laiiaioi 


Overall  Security  Class: 

Event  Number: 

Event  Name: 

Scheduling  Command:  Army 

Sponsoring  Organization :  Army 


•JELC  dates 
Sec.  Class: 


(U) 


US  Army-2011-0001 


Last  Modified: 


Test  Event  1 


(U) 


Start:  [▼]  End: 


<  r 


Inclusive  dates  - 


Sec,  Class:  (u) 


Start: 


py]  End:  [▼ 


I  ► 


Page  16 
3/17/2011 


How  it  Works  cont 


►I  Record  and  maintain  task  execution  results 


13  NDIA  SAMPLE  MISSION 
i  C3  ART  1.3.3  CONDUCT  TACTICAL  CONVOY 
Staff  Tasks 

I  ^  ART  6.4.2  MAINTAIN  CONSTANT  SITUATIONAL  AWARENESS 
■»-[3  Staff  Tasks 

P-  C3  ART  5.1 .4.1  MONITOR  SITUATION  OR  PROGRESS  OF  OPERATIONS 
o-  n  ART  5. 3. 5. 2.1  Collect  Friendly  Force  Information  Requirements 
*-  n  ART  5. 3. 5. 2. 2  Integrate  Intelligence  Products 

*-  C3  ART  5. 3. 5. 2. 4  Process  Relevant  Information  to  Create  a  Common  Operal 
o-  ART  5. 3. 5. 2. 5  Display  a  Common  Operational  Picture  Tailored  to  User  Ni 
o-  ART  5. 3. 5. 2. 6  Store  Relevant  Information 

-  Q  Subordinate  UnitTasks 
0  Command-Linked  Tasks 

f  n  Subordinate  UnitTasks 

i  [3  ART  2. 3. 3.1  CONDUCT  A  ROUTE  RECONNAISSANCE 

-  Q  StaffTasks 

p- 1~1  Subordinate  UnitTasks 

o-  □  ART  1 .6.1  OVERCOME  BARRIERS,  OBSTACLES,  AND  MINES 

o-  [3  Command-Linked  Tasks 
C3  ART  2.3.4  CONDUCT  SURVEILLANCE 
o-  C3  ART  6. 5. 3. 4  ESTABLISH  OBSERVATION  POSTS 
0  Command-Linked  Tasks 
7  O  ART  1.5.1  OCCUPY  AN  ASSEMBLY  AREA 
D  StaffTasks 

P 1~1  Subordinate  UnitTasks 

o-  ART  2. 3. 3. 3  CONDUCT  AN  AREA  RECONNAISSANCE 
0  Command-Linked  Tasks 
7  [3  ART  6. 5. 3. 3  ESTABLISH  PERIMETER  SECURITY 
I-  D  StaffTasks 
C3  Subordinate  UnitTasks 
I—  IQ  Command-Linked  Tasks 


■Mil  ART  7.5.1  9  ISOLATE  AN  ENEMY  FORCE 
i-  [3  StaffTasks 
f  n  Subordinate  UnitTasks 


.[lDTLjr'^~i^.1 ij:Li.tvl.ri.LliL^TjI'LLPijLCJ LL.L!.lj.li:jiC 


Mission  Task  f  Conditions  [  Standards-}  Notes  Observations  ||  Doctrine  *  [  TPAs  * 


M# 

Sec.  Class. 

Observed 

Scale 

Measure 

1 

<U) 

90min 

Time 

(U)  To  conduct  reconnaissance  of  obstacle 

2 

<U) 

1 20min 

Time 

(U)  To  complete  obstacle  clearing 

3 

<u> 

Number 

(U)  Of  friendly/neutral  casualties  caused  by  mine/IED  det... 

I.  Conditions  j  Standards  [  Issues 


Add... 


Details 


Remove... 
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How  it  Works  cont 


►I  Record  and  maintain  task  assessment  results 


NDIA  sample  mission 

f  C3  ART  1.3. 3  CONDUCT  TACTICAL  CONVOY 
|-|I3  Staff  Tasks 

^  [3  ART  6.4.2  MAINTAIN  CONSTANT  SITUATIONAL  AWARENESS 
9- [3  StaffTasks 

if-  C3  ART  5.1 .4.1  MONITOR  SITUATION  OR  PROGRESS  OF  OPERATIONS 
*-  [3  ART  5. 3. 5. 2.1  Collect  Friendly  Force  Information  Requirements 
C3  ART  5. 3. 5. 2. 2  Integrate  Intelligence  Products 

Q  ART  5. 3. 5. 2. 4  Process  Relevant  Information  to  Create  a  Common  Operal 
C3  ART  5. 3. 5. 2. 5  Display  a  Common  Operational  Picture  Tailored  to  User  Ni 
o-  [3  ART  5. 3. 5. 2. 6  Store  Relevant  Information 
0  Subordinate  UnitTasks 
0  Command-Linked  Tasks 
f  l~1  Subordinate  UnitTasks 

I  i  C3  ART  2. 3. 3.1  CONDUCT  A  ROUTE  RECONNAISSANCE 
D  StaffTasks 

9  n  Subordinate  UnitTasks 

[3  ART  1 .6.1  OVERCOME  BARRIERS,  OBSTACLES,  AND  MINES 
o-  C3  Command-Linked  Tasks 
9-  □  ART  2.3.4  CONDUCT  SURVEILLANCE 
e-  C3  ART  6. 5. 3. 4  ESTABLISH  OBSERVATION  POSTS 
0  Command-Linked  Tasks 
f  C3  ART  1 .5.1  OCCUPY  AN  ASSEMBLY  AREA 
I-  D  StaffTasks 
f  n  Subordinate  UnitTasks 

o-  [3  ART  2. 3. 3. 3  CONDUCT  AN  AREA  RECONNAISSANCE 
0  Command-Linked  Tasks 
f  [3  ART  6. 5. 3. 3  ESTABLISH  PERIMETER  SECURITY 
I-  D  StaffTasks 
o-  [3  Subordinate  UnitTasks 
Q  Command-Linked  Tasks 


■p—  ART  7.5.1  9  ISOLATE  AN  ENEMY  FORCE 
if-  C3  StaffTasks 

l~1  Subordinate  UnitTasks 
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Questions  to  Ask 


►I  To  properly  describe  mission  context  for  Systems  under  test 

►  What  is  the  source  scenario/vignette/CONOPS  for  the  mission? 

>  Is  it  the  same  CONOPS  used  to  develop  requirements  for  the  system? 

>  If  not,  why  not?  Valid  reasons  might  include  guidance  from  on  high,  changes 
in  assessment  of  current  and/or  future  operating  environment,  etc. 

►  Are  tasks  already  identified  in  one  or  more  of  the  requirements 
documents  (i.e.  FAA,  FNA,  ICD,  OMS-MP)? 

>  As  the  independent  evaluator,  are  you  satisfied  that  all  relevant  tasks  are 
included? 

>  Do  the  tasks  include  conditions  and  standards?  If  not,  where  do  they  come 


from? 


►  Are  the  tasks  decomposed  to  level  where  they  can  be  mapped  to  system 
attributes  and  functions? 

>  Are  they  linked  to  other  tasks  and  the  mission?  If  not,  how  will  you 
determine  and  justify  an  assessment  of  mission  impact? 
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Summary 


►I  Major  Points: 

►  Understanding  the  Mission  is  essential  to  MBT&E.  Selecting  the  right 
Mission  is  also  essential 

►  The  process  for  analyzing  and  specifying  the  Understanding  of  the 
Mission  using  a  common,  authoritative  language  is  well  defined  and 
doctrinally  based 

►  Automated  tools  are  available  to  assist  in  capturing,  maintaining, 
managing  and  sharing  results  of  Mission  Analysis  and  conversion  to  ATL 
tasks  as  well  as  other  T&E  related  functions 

►I  Recommendations/Way  Ahead: 

►  Adopt  and  codify  a  “how-to”  guide  for  specifying  Mission  Understanding 

►  Evaluate  and  select  existing  GOTS  and  COTS  tools  to  facilitate 
knowledge  capture 

►  Coordinate  and  collaborate  with  Joint  and  Service  requirement 
communities  (e.g.  JFCOM/JS  J7,  TRADOC  ARCIC,  TRAC,  etc)  to  clarify 
desired  format  for  Scenario/CONOPS  products 
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QUESTIONS? 


FOR  ADDITIONAL  INFORMATION  OR 
QUESTIONS  PLEASE  CONTACT: 

BRITT  BRAY 
Dynamics  Research  Corporation 
(785)  550-5573 
bbrav@drc.com 
Alt.  email:  britt.bray@us.army.mil 


Page  21 
3/17/2011 


Utilization  of  Modeling  and 
Simulation  for 
Networked  Waveform 
Characterization  and  Validation 


Scott  Rediger 
Rockwell  Collins 
ssredige@rockwellcollins.com 
(319)  295-1723 


Rockwell 

Collms 


Copyright  2011  Rockwell  Collins,  Inc. 
All  rights  reserved. 


March  8,  2011 


Rockwell 
Collins 

Agenda 

•  Discuss  Modeling  and  Simulation  used  for  Networked  Waveform 
Development  and  Validation 

-  What  is  a  Networked  Waveform? 

-  Why  is  simulation  required  for  Networked  Waveforms 

-  How  Modeling  and  Simulation  can  be  applied  and  utilized  through  the 
entire  product  lifecycle 

-  Prerequisites  for  using  Simulation 

-  Examples  of  lessons  learned  from  Networked  Waveform  Simulation 
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What  is  a  Networked  Waveform? 


•  Sometimes  referred  to  as  Mesh  Network  or  Mobile  Ad-Hoc  Network 

•  Self-configuring  network  of  nodes  connected  via  wireless  data  links 

-  Each  node  dynamically  adapts  to  evolving  network  topologies 

•  Network  protocols  ensure  that  all  nodes  are  kept  abreast  of  topology  updates 

•  Data  can  successfully  route  through  the  network  with  varying  numbers  of  hops 
depending  on  the  topology 

•  Nodes  are  free  to  physically  move  about  in  any  direction 

-  Nodes  can  be  on  land,  sea,  or  air 

•  Network  topology  changes  over  time  based  on: 

-  Each  node's  physical  location 

-  Vehicle/Aircraft  dynamics 

-  Node  configuration  changes 

-  Environmental  effects 
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Example  Network  Topology 


Irt  Operational  Map 


i'j  i 
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Example  Network  Topology 


©  Copyright  2011  Rockwell  Collins,  Inc, 
All  rights  reserved. 


5 


Rockwell 
Collins 

Why  is  Simulation  Required? 

•  The  number  of  variables  involved  in  a  networked  waveform  are  far  too 
many  for  static  analysis 

-  Vehicle/Aircraft  types 

-  Vehicle/Aircraft  dynamics 

-  Antenna  patterns  per  vehicle/aircraft 

•  including  polarization  and  shadowing 

-  Different  network  sizes 

-  Traffic  profiles 

-  Different  bandwidth  usage  profiles 

•  Networking  is  not  about  absolute  determinism,  but  rather  statistical 
probability 

-  Requires  repetitive  testing  to  characterize  a  network 

-  Requires  both: 

•  controlled  sequences  of  events 

•  random  sequences  of  events 
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Why  is  Simulation  Required? 

•  Testing  network  limits  is  impractical  with  real  hardware 

-  Maximum  number  of  nodes  (100+) 

-  Maximum  bandwidth  utilization  (90%  -  100%) 

•  With  limited  assets  available,  it  requires  unrealistic  loading  on  individual  nodes 


•  Validating  a  network  design  requires  different  types  of  testing: 

-  Repetitive  (Regression)  testing  with  fixed_  conditions  to  ensure  network 
behavior  is  deterministic  to  the  desired  degree 

-  Repetitive  (Regression)  testing  with  injected,  randomness  to  discover 
hidden  corner  conditions  and  network  heuristics 

-  Human  Gremlin  -  testing  with  an  eye  to  breaking  things 

•  Intentionally  stressing  network  in  ways  it  may  not  be  intended  to  be  used 

•  Ensuring  it  ends  up  in  a  known  state  and  recovers  under  all  conditions 

-  Monte  Carlo  style  testing 

•  Automated  running  of  tests  with  data  collection  and  analysis  to  determine 

boundary  conditions  and  statistical  probabilities  of  the  network 

- 1 - \ 

Testing  and  Characterization  of  a  Networked  Waveform  is  expensive 
and  requires  a  comprehensive  testing  strategy 
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What  comprises  a  Networked  Waveform  Simulator? 


•  A  network  simulator  facilitates  focused  testing  on  the  networking  layer 
of  a  waveform 

-  Simulates  targeted  non-networking  aspects  of  the  overall  system 


•  Well-defined  interfaces  allow 

Code-in-the-Loop  use  of  networking  layer 


Networking  to  SiS  interface 


ZZ  Upper  Layer  Applications 


Well  defined  interfaces  are  key  to  facilitating 
effective  Code-in-the-Loop  simulation 


Simulation  GUI  /  Scripting  Interface 


Simulation  Participant  i  Ground  Station 
Datalink  Applications 


Simulation  Datalink  Controller 


Networking  Layer 
Code-in-the-  Loop 


Applications 


Simulation  Radio  Interface 


Simulator  Interface 


Simulation  Core 


Time  Base 


SiS  Model 


Topographical 

Database 


Participant 

Navigation 

Policies 


Simulation  Engine 


©  Copyright  2011  Rockwell  Collins,  Inc. 
All  rights  reserved. 


8 


What  comprises  a  Networked  Waveform  Simulator? 


Simulator  Engine 


-  Provides  a  controlled,  synthetic  environment 

•  Topography  (physical  terrain,  obstructions) 

•  Node  Navigation  /  Mobility 

•  Signal  in  Space  Model 

-  Antenna  models 

-  Physics  of  waveform  modulation 

-  Propagation  delays  /  effects 

•  Time 

-  Simulation 

-  Allows  pausing  of  simulation  for  inspection 

-  Allows  simulation  to  run  slower  than 
real-time  as  model  fidelity  increases 

-  Allows  initial  simulation  to  focus  on 
networking  algorithms  themselves 
independent  of  real-time  constraints 


Simulation  GUI  /  Scripting  Interface 


Simulation  Participant  V  Ground  Station 
Datalink  Applications 


Simulation  Datalink  Controllef 
Applications 


& 


Networking  Layer 
Code-in-the-Loop 


Simulation  Radio  Interface 


Simulator  I  nterface 


Simulation  Core 


Tima  Base 


SiS  Model 


Topographical 

Database 


Participant 

Navigation 

Policies 


Simulation  Engine 
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Considerations  for  Simulation 


•  Before  you  ever  start,  you  should  be  able  to  answer  these  two 
questions: 

-  What  are  we  trying  to  measure? 

•  Simulation  Architecture  Requirements 

-  What  do  we  want  to  measure  in  the  future? 

•  Refine  Simulation  Architecture  Requirements 

•  Tradeoff  Criteria  Determination 


The  answers  to  these  two  questions  have  a 
significant  impact  on  the  total  cost  of  Simulation 


©  Copyright  2011  Rockwell  Collins,  Inc. 
All  rights  reserved. 


10 


Rockwell 
Collins 

Considerations  for  Simulation 

•  Determining  what  you  ARE_  NOT  simulating  is  almost  as  important  as 
determining  what  you  ARE_  simulating 

•  There  is  a  tradeoff  between  the  fidelity  of  the  model  and  the  hardware 
resources  required  to  keep  simulation  real-time 
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What  are  Some  Uses  of  Simulation  through  the 

Development  Lifecycle? 


•  Proof  of  Concept 

-  Determine  if  this  is  a  viable  network  design 

-  Determine  risks  in  proposals  (RFIs  /  RFPs) 

-  Perform  initial  trade  studies 

•  Determine  the  right  thresholds  and  objectives 


•  Network  Design  Validation 

-  As  the  simulation  model  is  matured,  better  assessments  can  be  made  for 
corner  conditions  and  design  constraints,  as  well  as  requirements  trades 

-  Prior  to  Hardware  being  built 

•  Real-time  porting  baseline  and  debugging  tool 

-  Once  real  hardware  is  available,  INTEGRATION  begins 

-  Simulation  provides  a  baseline  characterization  that  can  be  used  to 
diagnose  and  track  down  hardware  porting  issues 


©  Copyright  2011  Rockwell  Collins,  Inc. 
All  rights  reserved. 


12 


Rockwell 
Collins 

What  are  the  Uses  of  Simulation  through  the 

Development  Lifecycle? 

•  Traffic  Generator 

-  If  the  Simulation  Environment  is  designed  with  proper  hooks  in  place,  the 
simulated  nodes  can  generate  network  traffic 

•  Lab  bench  testing  of  real  hardware  in  a  loaded  condition 

•  Flight  testing  can  be  with  a  loaded  network  as  well 


•  Mission  Planning  Tool  /  Mission  Playback  Tool 

-  Growth  opportunity  to  enhance  the  simulation  model  fidelity  such  that 
missions  can  be  validated  via  simulation  before  any  aircraft  deploy 

•  Identify  network  choke  points  in  mission  plans 

•  Ensure  adequate  network  coverage  for  theater  of  operations 

•  Test  logistical  aspects  of  larger  networks 

-  Playback  allows  captured  flight  data  to  be  fed  into  Simulator 

•  Allows  refining  of  simulator  model  fidelity  by  comparing  actual  flight  data 
to  simulated  flight  data 
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Proof  of  Concept  Phase 

•  Focus 

-  Straw-man  fleshing  out  of  network  algorithms 

-  Discover  dynamic  aspects  and  corner  conditions 

-  Mitigation  of  High-Risk  Items 

-  Identification  of  key  strengths  and  limitations 


Network  Design 
Validation 


GIG 

Real-time  Porting 
Baseline 


Characteristics 

-  Low-Fidelity  physical  environment  modeling 

•  Basic  Signal  in  Space  model 

•  Basic  antenna  models 

•  Basic  topography  models  (maybe  even  2-D  vs.  3-D) 


Traffic  Generator 


i 


Mission  Planning 
Tool 


] 


•  Entire  Networking  solution  does  not  need  to  be  implemented  or  simulated 
-  Only  that  which  is  necessary  to  mitigate  high  risk  items 


•  Special  Considerations 

-  Is  this  throw  away  code? 
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Network  Design  Validation 


Proof  of  Concept 


•  Focus 

-  Full  development  and  demonstration  of  networking  algorithms  and 
services 

-  Code-in-the-Loop  simulation 

-  High-Fidelity  Signal  in  Space  model  (with  antenna  models) 

-  High-Fidelity  Topographical  models 

-  Uncovering  and  fixing  any  dynamic  aspects  and  corner  conditions 

-  Mitigation  of  as  many  risk  items  as  possible 

-  Documentation  of  Network  Design  (with  trades  documented) 


•  Characteristics 

-  Target  hardware  is  not  yet  available 


Mission  Planning 
Tool 


•  Special  Considerations 

-  While  waiting  for  real  hardware,  is  there  benefit  to  porting  to  an 
evaluation  board? 

-  Are  accurate  antenna  models  required  for  proving  network  design? 

-  How  accurate  does  our  model  need  to  be  to  adequately  validate  the 
network  design? 
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Real-time  Porting  Baseline 

•  Focus 

-  Simulation  is  used  as  a  performance  baseline  to  isolate 
porting  issues 

-  As  porting  bugs  are  fixed,  retest  fixes  in  simulation 
(Code-in-the-Loop) 

•  Characteristics 

-  Target  specific  simulation  scenarios  that  validate  issues 
found  in  porting  process 

-  Simulator  capabilities  are  not  further  refined  or  developed, 
but  rather  used  as  a  performance  reference  point 

-  Whenever  real-time  bugs  are  fixed,  the  simulator  code-in- 
the-loop  must  be  rebuilt  with  fixes  and  retested  in  the 
simulation  environment 


Proof  of  Concept 


Network  Design 
Validation 


Mission  Planning 
Tool 
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Traffic  Generator 

•  Focus 

-  Generating  network  traffic  (network  traffic  loading) 

-  Simulating  real  network  nodes  (network  tree  processing 
load) 

-  Assessing  real  hardware  performance  with  network  loading 

-  Make  any  simulator  real-time  performance  enhancements 
(if  necessary) 

•  Must  work  in  conjunction  with  real  hardware 

•  It  is  possible  that  fidelity  must  be  reduced  in  certain  simulation 
models  in  order  to  perform  in  real-time 


•  Characteristics 


Proof  of  Concept 


Network  Design 
Validation 


Real-time  Porting 
Baseline 


Mission  Planning 
Tool 


-  Focus  in  this  phase  is  not  networking  algorithms,  but  rather 
how  the  real  hardware  performs  in  various  loading  conditions 
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Mission  Planning  Tool 

•  Focus 

-  Enhancing  the  simulation  fidelity  to  the  point  where  it  can  be 
reliably  used  to  predict  mission  performance 

•  Enhanced  antenna  models 

•  Enhanced  topography  with  terrain  modeling  (will  slow  simulation 
way  down) 

•  Close  the  loop  on  Signal-in-Space  performance  with  real  flight 
testing  data 

-  Real-time  performance  is  not  the  focus,  accuracy  is  the  focus 


Proof  of  Concept 


G3G 

Network  Design 
Validation 


GIG 

Real-time  Porting 
Baseline 


Traffic  Generator 
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Simulator  Architecture 
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Simulator  Architecture  and  Features 


•  GUI  front-end 

•  Optional  scripting  capability 

-  Monte  Carlo  simulations 


•  Code-in-the-Loop  capability 


•  3-D  physical  model 

•  Earth  curvature  (WGS-84) 


•  Simulated  Time  Base 


•  Antenna  models 

•  Antenna  polarization 


•  Participant  Navigation 
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Simulator  Physical  Architecture 


Simulation  Control 
Scripted  Executable 
Framework 


Simulation  Control 
GUI  (Optional) 


Distributed  Simulation  Controller 


Windows  PC 
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Simulator  Physical  Architecture  -  Hardware-in-the-Loop 


Datalink  Controller 


RF 

Attenuator 

Control 


Simulation  Control 
(Script  or  GUI) 


RF 

Attenuator 

Control 


Poly-Node 
RF  Attenuator 


Simulation  PCs 


Legend 


Hardware/Software  In-the-Loop 
Simulation  Infrastructure 
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Traffic  Generation  /  Network  Loading 
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Software  considerations  with  simulation 


•  Start  with  simulation  in  mind 

-  Simulator  architecture  must  be  compatible  with  networking  code 

-  Concurrency  model  vs.  Software  Development  Plan 

•  Global/static  variables 

•  Threading  models 

•  Utilization  of  3rd  party  tools 

•  Abstraction  layers  to  enable  simulation 


•  Define  clear  interfaces  between  layers 

•  Make  sure  abstraction  layers  are  efficiently  implemented 

-  Many  of  them  are  high  iteration,  and  if  not  efficiently  done  can  negatively 
impact  the  final  code 

•  Iterative  Development  Cycles 

-  Simulation  model  is  not  effective  with  waterfall  development  model 
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Examples  of  Real  Simulation  Findings 

•  Efficient  slot  allocation  is  very  complex  and  can  lead  to  computationally 
intensive  calculations 

•  Early  trade  study  on  multiple  networking  modes  uncovered  the  need  for 
customized  link  quality  thresholds  that  were  dependent  on  mode  of  operation 

•  Three  and  four  hop  network  spans  do  not  happen  easily  -  like  water  finding  its 
level,  the  networking  layer  finds  a  shorter  path  before  humans  can  see  it 
visually 

•  Make-Before-Break  paradigm  for  healing  broken  data  paths  was  verified  to 
reduce  data  loss  and  was  weighed  against  the  temporary  increase  in 
bandwidth  required 
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March  16, 2011 
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Dover  NJ,  07801 
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I 


OoatlQHie 


>  What  was  our  Assignment 

>  The  Approach  to  the  Assignment 

>  model  Based  Systems  Engineering  (MBSE) 

>  Systems  Modeling  (SysML) 

>  Pillars  of  SysML 

>  Capturing  Requirements,  Behavior,  and  Structure  for  our 
assignment 

>  Capturing  Test  Information 

>  Other  Modeling  Activities 

>  Planning  Activities 

>  Lessons  Learned 


HPTi  Proprietary  Information 


The  Facility  m%€  the  Assignment 


Hardware  in  the  Loop  (HIL)  Facility 

>  Focus  on  testing  of  GPS-guided  precision  munitions 

>  Desiring  a  cost  effective  means  for  mitigating  risks 

>  Capable  of  performing  component  and  integrated 
component  tests  prior  to  gun  launch  testing 

Our  Assignment 

>  Capture  Stakeholder  Requirements 

>  Capture  System  Requirements 

>  Capture  Test  and  Evaluation  information  that  the  HIL 
Facility  offers 

>  Traceability  of  Test  and  Evaluation  information  to  the 
Requirements 


Information 


IJ  low  to  capture  the  information 

for  ©nr  assignment? 

I 

■  Asked  ourselves  how  to  best  accomplish  our 
assignment 

•  Desire  to  capture  Requirements,  System  Behaviors,  and 
Test  information  in  one  location  with  traceability 

■  Desire  to  involve  all  stakeholders  in  the  process  and 
develop  a  common  understanding  early  in  the  lifecycle 

•  Need  to  manage  project  risk 

•  Looked  to  a  Model  Based  Systems  Engineering 
Approach  to  help  achieve  this 

■  Focus  on  early  developmental  activities 
>  Scoping  the  system  of  interest 

Systems  Engineering  Approach 


HPTi  Proprietary  Information 
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Ml^SE  -  GeneraJ  G^Jiniition 


It  is  about  System  Modeling 

>  System  Model  is  a  cohesive,  unambiguous 
representation  of  what  the  System  Is  and  does. 

It  provides  a  description  of 


Past 


>  Requirements  and 

>  Technical  Solution  and 

❖  Operational  Scenarios 

❖  System  Behavior  (including  I/O) 

❖  Physical  Architecture  (Structure*  interfaces) 

❖  Dynamic  Simulation  (requires  “executable”  models) 

>  Verification  Procedures 

MUSE  is  used  to  produce  SE  products 


Document  centric 


O 


Future 


It  requires  a  Modeling  Language  that  is  computer 
interpretable 


Minimum 
Required  to 
Define  System 


HPTi  Proprietary  Information 


Descriptive  Modeling 


HPTi  Proprietary  Information 


SysilL  Overview 


■  General  Purpose 
Visual  Modeling 

>  Structure 

>  Behavior 

>  Requirements 

>  Parametric 

■  Supports: 
speeification,  analysis, 
design,  verification 
and  validation 


4  Piters  ef  SysML 


3.  Requirements 


U  LJ  iMiritnqi  U  adcdoOiflc'il  [J  5 


pm  Myftstemstncsfteflums  |3«tpie  Pwiwwi  Dayant)  ~) 
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4.  Parametrics 
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Capture  Capabil jties  of  the  HIL 


•  Eliciting  Threshold  and 
Objective  Capabilities 

•  Actors 

•  Use  Cases  (Goals) 

•  Used  to  review  with  team 

•  Helped  to  come  up  with 
stakeholder  requirements 
and  informally  trace 
behavior  to  requirements 

•  Looked  at  HIL  facility  as  a 
project 
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Capture  Structure  of  the  HIL 


•  Eliciting  Structure  of  the 
HIL 

•  What  is  part  of  the 
system 

•  What  is  outside  of 
system  that  interacts 
with  our  system 

•  Logical  Abstraction  of 
“things”  that  may  end  up 
being: 

•  Physical  Equipment 

•  Software 

•  Information  (e.g. 
documented 
procedures/enabling 
products) 


bdd  [Model]  Data[  HIL  Hierarchy  ]J 
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Capture  iiehaviw  of  the  HIL 


•  Eliciting  Behaviors  of  the 
HIL 

•  Could  use  Activity, 
Sequence,  and/or  State 
Diagrams 

•  Can  look  at  from  a 
domain  perspective 
(which  we  did  here) 

•  Here  we  elicit  the 
actions  for  testing  a 
weapon  (which  may  or 
may  not  be  tied  to  a 
specific  capability) 


activity  Domain  ActK/ities  [  [^j  Domain  ActK/ities  ]  | 
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Sco  jj  &  behavior  off  the  HIL 


•  Scope  Behaviors  of  the  HIL 

•  Used  the  activity 
diagrams  to  review 
actions  of  a  test 

•  Next,  it  helped  us 
decide  what  is  part  of 
the  system  and  what  is 
outside  the  system  (i.e. 
allocation  of  behavior  to 
structure  in  this  case) 
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Capture  R  9  tjuj foments  mi  the  HIL 


•  Capture  Requirements  of 
the  HIL 

•  This  was  going  on  in 
parallel  with  capturing 
the  capabilities, 
structure,  and  behavior 

•  Can  be  done  within  a 
modeling  tool, 
requirements 
management  tool,  or 
both 

•  Relationships  between 
the  requirements  and 
other  model  elements 
can  be  captured 


System  Requirements  in  a 
requirements  management  tool  »> 


IP  g|;<r.  ace.  itrfTwu 

^ 1  MIL  Program  Requirements  at  System  of  Interest  Level 
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1.1  The  system  shall ...  1  (system  level  requirement) 

■  1.2  The  system  shall  ...  2  (system  level  requirement) 
r  1.3  The  system  shell ...  3  (system  level  requirement) 

1.4  The  system  shall ...  4  (system  level  requirement)  (O) 

2  HIL  Facility  Requirements  at  System  of  Interest  Level 

2.1  The  system  shall ...  S  (system  level  requirement! 

2.2  The  system  shall  ...  6  (system  level  requirement) 

2.3  The  system  shall ...  7  (system  level  requirement] 

2.4  The  eyetem  shall ...  9  (ayatem  level  requirement) 

3  HIL  Test  Requirements  at  System  of  interest  Level 

}  3.1  The  system  shall ...  9  (system  level  requirement) 

'  3.2  The  system  shall ...  10  (system  level  requirement! 

1 3.3  The  system  shall ...  11  (system  level  requirement) 

3.4  The  system  shall ...  12  (system  level  requirement) 

4  HIL  Simulation  Requirements  at  System  of  Interest  Level 
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5  hil  Reporting  Requirements  at  System  of  Interest  Level 
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4.4  The  eyetem  shell ...  16  (ayatem  level  requirement)  *  Tut  )6 
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req  [Model]  Data]  Use  Case  Refines  ]  I 


« require  merits 

1.1  The  HIL  facility  shall  ...a 


{Id  =  "4", 

Text=  "1.1  The  HIL  facility  shall ...  a"} 


Q  I 


«  refine  » 
fr  — 
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Capture  R  9  tjuj foments  ©ff  the  HIL 


•  Capture  Requirements  of  the 
HIL 

•  A  trace  view  may  be  more 
appropriate  and 
manageable  for  large 
projects  than  a  diagram 

•  A  trace  view  can  be 
exported  to  a  deliverable  or 
format  that  can  be  used 
elsewhere  (e.g.  imported 
into  a  spreadsheet  or 
requirements  management 
tool). 

•  Some  tools  provide  tables 
that  would  allow  you  to 
managed  requirements 
within  the  MBSE  tool  (if 
desired). 


B-Q  Section  1  SYS 

•  Q1  15  1.1  The  system  shall ...  1  (system  level  requirement) 

•  CB  27  1 .2  The  system  shall ...  2  (system  level  requirement) 

•  CB  12  1.3  The  system  shall ...  3  (system  level  requirement) 

US  13  1.4  The  system  shall ...  4  (system  level  requirement)  (O) 
$--C3  Section  2  SYS 

■  -  CB  14  2.1  The  system  shall ...  5  (system  level  requirement) 

CB  16  2.2  The  system  shall ...  6  (system  level  requirement) 

....  qj  17  2.3  The  system  shall ...  7  (system  level  requirement) 

•••■  CB  18  2.4  The  system  shall ...  8  (system  level  requirement) 
Section  3  SYS 

|  CB  19  3.1  The  system  shall ...  9  (system  level  requirement) 
i  20  3.2  The  system  shall ...  10  (system  level  requirement) 

I  •  CB  213.3  The  system  shall ...  1 1  (system  level  requirement) 

L.  CB  22  3.4  The  system  shall ...  12  (system  level  requirement) 
Section  4  SYS 

-  CB  23  4. 1  The  system  shall ...  13  (system  level  requirement) 

CB  24  4.2  The  system  shall , , .  14  (system  level  requirement) 

■•••  CB  25  4.3  The  system  shall ...  15  (system  level  requirement) 

....  CB  26  4.4  The  system  shall ...  16  (system  level  requirement) 
S-D  Section  5  SYS 

L.  CB  28  5. 1  The  system  shall ...  17  (system  level  requirement) 

]••••  CB  29  5.2  The  system  shall ...  18  (system  level  requirement) 
i--CB  30  5.3  The  system  shall ...  19  (system  level  requirement) 
L.CB  31  5.4  The  system  shall ...  20  (system  level  requirement) 
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Capture  Verification  lnformsti@n 


•  Capture  Verification 
Information  for  the  HIL 

•  Assignment  was  also  to 
capture  how  the  system 
requirements  were 
going  to  be  verified. 

•  MBSE  can  capture  that 
information  (e.g.  relating 
verification  to 
requirements). 

•  This  can  be  captured 
and  displayed  in 
requirements  diagrams, 
trace  views,  and 
behavior  diagrams). 


req  [Package]  Test  Case  Package  [  Test  Diagram  for  Section  5  of  System  Requirements  jJ 


.•requirement*  c 

5.1  The  system  shall ...  17  (system  level  requirement) 


{Id  =  '28". 

Text  =  "5.1  The  system  shall ...  17  (system  level  requirement)"} 


•Test  Case* 

Test  Case  A 


..requirement*  c 

5.2  The  system  shall ...  18  (system  level  requirement) 


{Id  =  '29". 

Text=  "5.2  The  system  shall ...  18  (system  level  requirement)"} 


•Test  Case* 

Test  Case  B 


•requirement*  c 

5.3  The  system  shall ...  19  (system  level  requirement) 


{Id  =  '30", 

Texts  "5.3  The  system  shall ...  19  (system  level  requirement)"} 


•Test  Case* 

Test  Case  C 


•requirement*  c 

5.4  The  system  shall ...  20  (system  level  requirement) 


{ld=  '31". 

Texts  "5.4  The  system  shall ...  20  (system  level  requirement)"} 
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C®jj>tnrinf  Parametrics 


•  Capturing  equation  data  for 
your  system  of  interest 

•  Interface  with  solvers  to 
solve  your  equations 

•  Can  create  instances  to 
look  at  different  possible 
solutions  (e.g.  trade 
comparisons) 

•  Some  examples  of  possible 
use:  timeline  analysis, 
failure  analysis,  reliability 
analysis,  budgeting  (e.g. 
weight,  cost),  aeroballistics 
model,  optimize  test  set, 
model  risk 


bdd  [Package)  Addition!  AdditionBDD 


par  [Block)  Gamma  [  Gamma 


bdd  [Package]  AdditbnlnstanceOI  [  Instanced  ]} 


*bbck»  q 

alphaOl  :  Alpha 

a  ="15.0" 


«  block*  | — | 

BetaOl  :  Beta 

b=  '20.0" 


«bbck» 


al  =alpha01 
bl  =  BetaOl 
c="35.0" 


□ 
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C®jj>tnrinf  Parametrics 


•  Simple  example  here  is  for 
a  weight  budget. 

•  The  data  for  the  equation  is 
gathered  in  the  block 
definition  diagram. 

•  The  “wiring”  together  of 
weight  equation  is  done 
within  a  parametric 
diagram. 

•  The  data  can  now  be 
analyzed  (which  may  mean 
interaction  with  a  plug-in  to 
the  MBSE  tool  that  serves  a 
equation  solver). 
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G®]jtnring  Parametrics 


•  For  our  HIL  task 
assignment,  we  did  some 
capturing  of  parametric 
data  (informal). 

•  Interfaced  with  System 
Analysis  team  to  explain 
the  HIL  testing  related  to 
the  simulated  projectile 
flight  information. 

•  The  diagrams  to  the  right  is 
a  high  level  abstraction  of 
that  information 
(representative  example). 


bdd  [Package]  Block  Diagrams  [Input  Outputs  of  HIL]  ~J 


Fire  Mission  Data 
MET  Data 
Hardware  Under  Test 
Aeroballistics  Data 
GPS  Performance 
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Mo  del  AnnniMon  and  Execution 


•  MBSE  tools  can  be  used  to  animate/execute  behavior  of  your 


■e 

•t 


*  Executing  an  Activity  Diagram 

*  Executing  a  State  Machine  Diagram 

*  Executing  a  Sequence  Diagram 

Model  animation  can  help  with  gap  analysis 

Model  animation  identify  interfaces  within  your  system  and 
domain 

Model  animation  can  be  used  to  prototype  your  system  (or 
prototype  different  solutions/alternatives) 

An  executable  model  provides  the  potential  to  auto-generate 
useful  model  artifacts 
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Phummg  Consu  derations 


l — ii — *  i  ; 

•  Scoping  the  effort  (end  where  modeling  fits  in  for  specific 
project) 

•  Need  e  MBSE  process  to  follow  (an  approach) 

•  Common  Modeling  Language  (e.g.  SysML,  UML) 

•  A  Modeling  Tool  to  capture  the  information 

•  Who  is  going  to  model  the  information  (and  be  able  to  convey 
the  information  to  the  reviewers  who  aren’t  expected  to  be 
system  modelers  themselves) 

•  Who  is  going  to  review  the  information  (impacts  the  scoping 
of  the  effort  as  well) 
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Conclusions/Lessons  L-jcirn-jil 


•  Developed  a  common  understanding  of  our  system  and 
what  we  needed  to  verify 

•  Assisted  in  defining  and  confirming:  capabilities, 
requirements,  structure,  interfaces,  and  test  information 

«  Formally  documented  the  system  and  related  verification 
information 

•  Didn’t  cause  extra  work  (was  part  of  the  work;  modeling 
assisted  in  delivering  on  schedule  and  quality  work) 

•  Provided  confidence  to  leadership  that  the  project  was 
meeting  requirements  and  being  verified 
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Agenda 


Various  Approaches  to  Testing  Multiple  Factors 

What  makes  Design  of  Experiments  so  special? 

Using  DOE  to  build  transfer  functions  in  DT&E 

Critical  Parameter  Management:  linking  the  functions 
together 

High  Throughput  Testing  in  OT&E 
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Web-Based  Test  Scenario 
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Performance  Tuning  Terminology 


Factors/Inputs 

(X’s) 

Levels 

(Choices) 

Performance/Outputs 

(Y’s) 

CPU  Type 

Itanium,  Xeon 

#  home  page  loads/sec 

CPU  Speed 

1  GHz,  2.5  GHz 

Cost 

RAM  Amount 

256  MB,  1.5  GB 

HD  Size 

50  GB,  500  GB 

VM 

J2EE,  .NET 

OS 

Windows,  Linux 

Which  factors  are  important?  Which  are  not? 

Which  combination  of  factor  choices  will  maximize  performance? 
How  do  you  know  for  sure?  Show  me  the  data. 
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Approaches  to  Testing  Multiple  Factors 


•  Traditional  Approaches 

•  One  Factor  at  a  Time  (OFAT) 

•  Oracle  (Best  Guess) 

•  All  possible  combinations  (full  factorial) 

•  Modern  Approach 

•  Statistically  designed  experiments  (DOE) ...  full 
factorial  plus  other  selected  DOE  designs, 
depending  on  the  situation 


©2011  Air  Academy  Associates,  LLC.  Do  Not  Reproduce.  Page  4 


What  is  a  Designed  Experiment? 


Purposeful  changes  of  the  inputs  (factors)  in  order  to  observe 
corresponding  changes  in  the  output  (response). 


x, 


x. 


Inputs  x. 


x. 


* 

* 


PROCESS 


Yi 


Outputs 
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Run 

X,  X2  X3  X4 

CM 

>- 

T— 

>- 

Y 

Sy 

1 

2 

3 
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DOE  Helps  Determine  How  Inputs  Affect 

Outputs 


i) 

Factor  A  affects  the  average  of  y 

y 

ii) 

Factor  B  affects  the  standard  deviation  of  y 

ABi 

y 

iii) 

Factor  C  affects  the  average  and  the 

f 

standard  deviation  of  y 

iv) 

Mo, 

cy^\  /  \ 

y 

Air^ 

Factor  D  has  no  effect  on  y 

Academy 
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y 
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Evaluating  the  Effects  of  Variables  on  Y 


We  don’t  want  this: 


How  do  we  obtain  this  independence  of  variables? 
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Statistically  Designed  Experiments  (DOE): 

Orthogonal  or  Nearly  Orthogonal  Designs 


►  Taguchi  Designs 


FULL  FACTORIALS  (for  small  numbers  of  factors) 

FRACTIONAL  FACTORIALS 
PLACKETT  -  BURMAN 
LATIN  SQUARES 
HADAMARD  MATRICES 
BOX  -  BEHNKEN  DESIGNS 
CENTRAL  COMPOSITE  DESIGNS 
NEARLY  ORTHOGONAL  LATIN  HYPERCUBE  DESIGNS 

SIMPLE  DEFINITION  OF  TWO-LEVEL 
ORTHOGONAL  DESIGNS 


Run 


Academy 

Associates 


Actual  Settings 

(5,10)  (70,90) 

A:  Time  B:  Temp 

(100,200) 

C:  Press 

(A) 

Time 

Coded  Matrix 

(B) 

Temp 

(C) 

Press 

1 

5 

70 

100 

-1 

-1 

-1 

2 

5 

70 

200 

-1 

-1 

+1 

3 

5 

90 

100 

-1 

+  1 

-1 

4 

5 

90 

200 

-1 

+  1 

+1 

5 

10 

70 

100 

+  1 

-1 

-1 

6 

10 

70 

200 

+  1 

-1 

+1 

7 

10 

90 

100 

+  1 

+  1 

-1 

8 

10 

90 

200 

+  1 

+  1 

+1 

Responses 
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What  Makes  DOE  so  Powerful? 
(Orthogonality:  both  vertical  and  horizontal  balance) 

A  Full  Factorial  Design  for  3  Factors  A,  B,  and  C,  Each  at  2  levels: 

Run 

A 

B  C 

AB 

AC  BC 

ABC 

1 

- 

- 

+ 

+  + 

- 

2 

- 

+ 

+ 

- 

+ 

3 

- 

+ 

- 

+ 

+ 

4 

- 

+  + 

- 

+ 

- 

5 

+ 

- 

- 

+ 

+ 

6 

+ 

+ 

- 

+ 

- 

7 

+ 

+ 

+ 

- 

- 

8 

+ 

+  + 

+ 

+  + 

+ 
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Design  of  Experiments  (DOE) 

•  An  optimal  data  collection  methodology 

•  “Interrogates”  the  process 

•  Used  to  identify  important  relationships 
between  input  and  output  factors 

•  Identifies  important  interactions  between 
process  variables 

•  Can  be  used  to  optimize  a  process 

•  Changes  “I  think”  to  “I  know” 
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Google  on  DOE 

(quotes*  from  Daryl  Pregibon,  Google  Engineer) 


“From  a  user’s  perspective,  a  query  was  submitted  and  results 
appear.  From  Google’s  perspective,  the  user  has  provided  an 
opportunity  to  test  something.  What  can  we  test?  Well,  there  is  so 
much  to  test  that  we  have  an  Experiment  Council  that  vets 
experiment  proposals  and  quickly  approves  those  that  pass 
muster.” 

“  We  evangelize  experimentation  to  the  extent  that  we  provide  a 
mechanism  for  advertisers  to  run  their  own  experiments. 

. . .  allows  an  advertiser  to  run  a  (full)  factorial  experiment  on  its 
web  page.  Advertisers  can  explore  layout  and  content 
alternatives  while  Google  randomly  directs  queries  to  the 
resulting  treatment  combinations.  Simple  analysis  of  click  and 
conversion  rates  allows  advertisers  to  explore  a  range  of 
alternatives  and  their  effect  on  user  awareness  and  interest.” 

*  Taken  From:  Statistics  @  Google  in  Amstat  News,  May  2011 
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Transfer  Function:  A  Key  DT  and  OT  Concept 


r 

Parameters 
or  Factors 
that  ^ 
Influence 
the  CTP, 
MOE,  or 
MOP  ^ 


S  =f2(x1,x2,  x3) 
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Where  does  the  transfer  function  come  from? 

•  Exact  transfer  function 

•  Approximations 

-  DOE 

-  Historical  Data  Analysis 

-  Simulation 
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Exact  Transfer  Functions 


Engineering  Relationships 
-  V=  IR 


F  =  ma 


9V  -r 


X 


r 


£  < 


v 


> 

> 

> 

> 

> 

> 

Where 


N 

I 

r 

t 

x 

H 


The  equation  for  current  (I)  through 
this  DC  circuit  is  defined  by: 


/  = 


V  V(R,  +  R2) 


^■R2  R,  R2 


Rj  +  R2 


The  equation  for  magnetic  force  at  a  distance 
X  from  the  center  of  a  solenoid  is: 


H  =  — 

21 


.5£  +  x 


+ 


.5£  -x 


^r2  +  (.5^  +  x)2  ^r2  +  (.5£-x) 


total  number  of  turns  of  wire  in  the  solenoid 
current  in  the  wire,  in  amperes 
radius  of  helix  (solenoid),  in  cm 
length  of  the  helix  (solenoid),  in  cm 
distance  from  center  of  helix  (solenoid),  in  cm 
magnetizing  force,  in  amperes  per  centimeter 
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Value  Delivery:  Reducing  Time  to  Market  for 

New  Technologies 


INPUT 


Pitch  <) 

(0,  15,  30) 

Roll  <) 

(0,  15,  30)  . 

W1F<) 

(-15, 

0,  15) 

W2F  <) 

(-15, 

0,  15) 

W3F  <) 

(-15, 

0,  15)  . 

Modeling  Flight 
Characteristics 
of  New  3-Wing 
Aircraft 


OUTPUT 


Six  Aero- 


Characteristics 


Total  #  of  Combinations  =  35  =  243 
Central  Composite  Design:  n  =  30 


Patent  Holder:  Dr.  Bert  Silich 
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Aircraft  Equations 


II 

_i 

o 

.233  +  .008(P)2  +  .255(P)  +  .012(R)  -  .043(WD1)  -  .117(WD2)  +  .185(WD3)  +  .010(P)(WD3)  - 
.042(R)(WD1)  +  .035(R)(WD2)  +  .016(R)(WD3)  +  .010(P)(R)  -  .003(WD1)(WD2)  - 
.006(WD1)(WD3) 

II 

o 

o 

.058  +  .016(P)2  +  .028(P)  -  .004(WD1)  -  .013(WD2)  +  .013(WD3)  +  .002(P)(R)  -  .004(P)(WD1) 

-  .009(P)(WD2)  +  .016(P)(WD3)  -  .004(R)(WD1)  +  .003(R)(WD2)  +  .020(WD1)2  +  .017(WD2)2 
+  .021  (WD3)2 

o 

-< 

II 

-.006(P)  -  .006(R)  +  .169(WD1)  -  .121(WD2)  -  .063(WD3)  -  .004(P)(R)  +  .008(P)(WD1)  - 
.006(P)(WD2)  -  .008(P)(WD3)  -  .012(R)(WD1)  -  .029(R)(WD2)  +  .048(R)(WD3)  -  .008(WD1)2 

f 

Cm  ” 

.023  -  .008(P)2  +  .004(P)  -  .007(R)  +  .024(WD1)  +  .066(WD2)  -  .099(WD3)  -  .006(P)(R)  + 
.002(P)(WD2)  -  .005(P)(WD3)  +  .023(R)(WD1)  -  .019(R)(WD2)  -  .007(R)(WD3)  +  .007(WD1)2 
-  .008(WD2)2  +  .002(WD1)(WD2)  +  .002(WD1)(WD3) 

j\ 

II 

>- 

o 

.001  (P)  +  .001  (R)  -  .050(WD1)  +  .029(WD2)  +  .012(WD3)  +  .001(P)(R)  -  .005(P)(WD1)  - 
.004(P)(WD2)  -  .004(P)(WD3)  +  .003(R)(WD1)  +  .008(R)(WD2)  -  .013(R)(WD3)  +  .004(WD1)2 
+  .003(WD2)2  -  .005(WD3)2 
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ce  = 

.003(P)  +  .035(WD1)  +  .048(WD2)  +  .051  (WD3)  -  .003(R)(WD3)  +  .003(P)(R)  -  .005(P)(WD1) 

+  .005(P)(WD2)  +  .006(P)(WD3)  +  .002(R)(WD1) 
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Fusing  Titanium  and  Cobalt-Chrome 


I 


Courtesy  Rai  Chowdhary 
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DOE  “Market  Research”  Example  (cont.) 


Question:  Choose  the  best  design  for  evaluating  this  scenario 


Answer: 


L18  design  with  attributes  A  -  H  in  the  inner  array  and 
factors  J,  K,  and  L  in  the  outer  array,  resembling  an 
L18  robust  design,  as  shown  below: 


Academy 

Associates 

Simplify,  Perfect,  Innovate 


Run* 


1 

2 

3 

4 

5 

6 

7 

8 

9 

10 
11 
12 

13 

14 

15 

16 

17 

18 


A  B  C  D 


F  G  H 


o 

+ 

0 

+ 

0 

+ 


0 

0 

+ 


L 

K 

J 


18  different  product  profiles 


+  -  + 
+  + 


+ 

+  + 

+  + 


yi  y2  y3  y4  y5  y6  y7  y8 


Segmentation  of  the  population  or 
Respondent  Profiles 
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© 


External  Market  Factors 
(Local  Labor  Market  Conditions) 


Local  Unemployment  Rate 


Local  Employment  Alternatives 


Company’s  Market  Share 


Organizational  Characteristics 
and  Practices 


Supervisor  Stability 


Lateral  /  Upward  Mobility 


Layoff  Climate 


(D  Employee  Attributes 


Time  Since  Last  Promotion 


Education  Level 


Job  Stability  History 


Process  of 
Deciding  to 
Stay  /  Leave 


Turnover  Rate 


'"Adapted  from  Harvard  Business  Review  article  on  Boston  Fleet  Bank,  April  2004,  pp  116-125 
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The  Value  of  Transfer  Functions 


■  Provide  a  simple  and  compact  wav  of  understanding 
relationships  between  performance  measures  or  response 
variables  (y’s)  and  the  factors  (x’s)  that  influence  them. 

■  Allow  for  the  prediction  of  the  response  variable  (y),  with 
associated  risk  levels,  before  any  change  in  the  product  or 
process  is  made. 

■  Allow  for  the  assessment  of  process  or  product  capability  in 
the  presence  of  uncontrolled  variation  or  noise. 

■  Allow  the  very  quick  manipulation  of  complex  systems  using  a 
meta-model  derived  from  a  simulator  via  DOE. 

■  Provide  a  very  easy  wav  to  optimize  performance  via  robust  or 
parameter  design  and  tolerance  allocation. 

■  Make  sensitivity  analysis  easy  and  straightforward. 

■  Greatly  enhance  one’s  knowledge  of  a  product  or  process. 

■  In  general,  they  are  the  gateway  to  systematic  innovation. 

■  Provide  a  meaningful  metric  for  the  maturity  in  DFSS  for  any 
organization. 
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Critical  Parameter  Management  and  COIs 


-A Critical  Operational  Issue  (COI)  is  linked  to  operational  effectiveness  and 
suitability. 

-  It  is  typically  phrased  as  a  question,  e.g., 

Will  the  system  detect  the  threat  in  a  combat  environment  at 
adequate  range  to  allow  for  successful  engagement? 


y2  (engagement) 


xl  x2  x3  (ranges)  x4  (threat  type)  x5  x6 
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CPM  is  a  systems  engineering  best  practice  that  is  extremely  useful  in 
managing,  analyzing,  and  reporting  technical  product  performance.  It  is 
also  very  useful  in  decomposing  COIs  and  developing  linkages  between 
measures  and  task  capabilities/system  attributes. 


“The  System  Can....'1 


-AAA 

h 

AAA 

m 


a  AAA 

A  S 


Vs 
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Multiple  Response  Optimization 
Simulation*  Example 


Cross  Support 

Diameter  (2-4) 

Design  Cost  (<  300) 

Vertical  Support 

Diameter  (2-4) 

Vehicle 

Impact 

Rib  Compression  (<  11) 

Safety 

f 

Box  Diameter  (2-4) 

Process 

T 

Triaxial  Acceleration  (<  15) 

Beam  Angle  (35-60) 

AIR* 

Academy 

Associates 
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*  From  SimWare  Pro  by  Philip  Mayfield  and  Digital  Computations 
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Capability  Prior  to  Optimization 


Input  Controls 


Control  Set  1 , 


G.OOO 


3 
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Cross  Support  Diameter  (2  to 
41 


TOO  3  !200 

Vertical  Support  Diameter  [2  to 
A) 


vj  |40.00  ~^j 

Box  Diameter  (2  to  A)  Beam  Angle  (35  to  60) 


Simplify,  Perfect,  Innovate 


Multiple  Response  Optimization  (cont.) 

Capability  After  Optimization 


.poll  Cpk 


D  rib. compression  Cpk 


triaxial_accele  ration  Cpk  - 


±!  Cl 


270  DO  290-QQ  310  00  330  00 


<  > 


eao  0  50  iD  2o  ioso 


<  > 


Input  Controls 


Control  Set  1 


2.270 


Cross  Support  Uiameter  (2  to 
A) 


|3.78  3  14.00 

Vertical  Support  Uiameter  ( 'Z  to 
A) 


3  3 

Box  Diameter  (2  to  A)  Beam  Angle  (35  to  60) 
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■input  Controls 


Control  Sell 


(500 


Academy 
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472.727272727273  581.818181818182  690909090909091 


909.090909090909 


|S  Current  Data 
n  =  159 
Mean  =  681.9 
Standard  Deviation  =  141.6 
LSL  =  700 
USL  =  900 
Cp  =  0.2354 
Cpk  =  -0.0427 
DPM  =  612,650 
Sig  Cap  =  1.214 


SiU 


~j]  |12j00 


3  |145 


3 


Plug  Pressure  (20  to  50) 


Bellow  Pressure  (1 0  to  20) 


Ball  Valve  Pressure  (1 00  to  200) 
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70  '^r  100 

Water  Temp  (Expensive  to  Contrd) 
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-  Current  Data 
n  =  150 
Mean  =  802 

Standard  Deviation  =  21.58 
LSL  =  700 
USL  =  900 
Cp- 1.546 
Cpk  =  1.518 
DPM  =  3.829 
Sig  Cap  =  5.975 
0  Memorized  Data 
n  =  159 
Mean  =  681.9 
Standard  Deviation  =  141.6 
LSL  =  700 
USL  =  900 
Cp  =  0.2354 
Cpk  =  -0.0427 
DPM  =  612,650 
Sig  Cap  =  1.214 


Input  Controls 

Control  Set  1 


85 


3 


Plug  Pressure  (20  to  50) 


Bellow  Pressure  (1 0  to  20) 


Ball  Valve  Pressure  (1 00  to  200) 


Water  Temp  (Expensive  to  Control) 
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Introduction  to  High  Throughput  Testing 

(HTT) 

•  A  recently  developed  technique  based  on  combinatorics 

•  Used  to  test  myriad  combinations  of  many  factors  (typically  qualitative) 
where  the  factors  could  have  many  levels 

•  Uses  a  minimum  number  of  runs  or  combinations  to  do  this 

•  Software  (e.g.,  ProTest)  is  needed  to  select  the  minimal  subset  of  all  possible 

combinations  to  be  tested  so  that  all  2-way  combinations  are  tested. 

•  HTT  is  not  a  DOE  technique,  although  the  terminology  is  similar 

•  A  run  or  row  in  an  HTT  matrix  is,  like  DOE,  a  combination  of  different  factor 

f 

levels  which,  after  being  tested,  will  result  in  a  successful  or  failed  run 

•  HTT  has  its  origins  in  the  pharmaceutical  business  where  in  drug  discovery 
many  chemical  compounds  are  combined  together  (combinatorial  chemistry) 
at  many  different  strengths  to  try  to  produce  a  reaction. 

AIR* 

Academy 

Associates 

Simplify,  Perfect,  Innovate 

•  Other  industries  are  now  using  HTT,  e.g.,  software  testing,  materials 

discovery,  integration  and  functionality  testing  (see  example  on  next  page). 
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Submarine  Threat  Detection  Example 


■Suppose  we  want  to  perform  a  verification  test  with  the  following  7 
input  factors  (with  their  respective  settings): 

•Submarine  Type  (SI ,  S2,  S3) 

•Ocean  Depth  (Shallow,  Deep,  Very  Deep) 

•Sonar  Type  (Active,  Passive) 

•Target  Depth  (Surface,  Shallow,  Deep,  Very  Deep) 

•Sea  Bottom  (Rock,  Sand,  Mud) 

•Control  Mode  (Autonomous,  Manual) 

•Ocean  Current  (Strong,  Moderate,  Minimal) 

■All  possible  combinations  would  involve  how  many  runs  in  the  test? 

■If  we  were  interested  in  testing  all  pairs  only,  how  many  runs  would 
be  in  the  test?  Pro  Test  generated  the  following  test  matrix. 
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Submarine  Threat  Detection  Example  (cont.) 


The  following  15  test  cases  will  test  all  pairwise  combinations. 


Factor A 

Factor B 

Factor C 

Factor D 

Factor E 

Factor F 

Factor G 

Factor 

Name 

Submarine  Type 

Ocean  Depth 

Sonar  Type 

Target  Depth 

Sea  Bottom 

Control  Mode 

Ocean  Current 

Case  1 

S3 

Deep 

Passive 

Very  Deep 

Mud 

Manual 

Minimal 

Case  2 

SI 

Very  Deep 

Passive 

Surface 

Rock 

Autonomous 

Strong 

Case  3 

S2 

Shallow 

Active 

Shallow 

Rock 

Manual 

Moderate 

Case  4 

S2 

Deep 

Passive 

Deep 

Sand 

Autonomous 

Moderate 

Case  5 

SI 

Shallow 

Active 

Surface 

Sand 

Manual 

Minimal 

Case  6 

SI 

Very  Deep 

Passive 

Shallow 

Mud 

Autonomous 

Minimal 

Case  7 

S3 

Very  Deep 

Active 

Deep 

Mud 

Manual 

Strong 

Case  8 

S2 

Very  Deep 

Active 

Very  Deep 

Sand 

Autonomous 

Strong 

Case  9 

S3 

Shallow 

Passive 

Shallow 

Mud 

Autonomous 

Strong 

Case  1 0 

S3 

Deep 

Active 

Surface 

Rock 

Manual 

Moderate 

Case  11 

SI 

Shallow 

Active 

Deep 

Rock 

Autonomous 

Minimal 

Case  1 2 

SI 

Deep 

Passive 

Very  Deep 

Rock 

Manual 

Moderate 

Case  13 

S2 

Very  Deep 

Active 

Surface 

Mud 

Autonomous 

Moderate 

Case  1 4 

S3 

Deep 

Active 

Shallow 

Sand 

Manual 

Strong 

Case  15 

S2 

Shallow 

Active 

Very  Deep 

Rock 

Manual 

Minimal 
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Command  &  Control  Test  Example 

(15  factors  each  at  various  levels) 

Total  Combinations:  20,155,392 

Variable  or  Factor 

Levels 

(#  of  levels) 

Mission  Snapshots 

Entry,  Operations,  Consolidation 

(3) 

Network  Size 

10  Nodes,  50  Nodes,  100  Nodes 

(3) 

Network  Loading 

Nominal,  2X,  4X 

(3) 

Movement  Posture 

ATH,  OTM1,  OTM2 

(3) 

SATCOM  Band 

Ku,  Ka,  Combo 

(3) 

SATCOM  Look  Angle 

0,  45,  75 

(3) 

Link  Degradation 

0%,  5%,  10%,  20% 

(4) 

Node  Degradation 

0%,  5%,  10%,  20% 

(4) 

EW 

None,  Terrestrial,  GPS 

(3) 

Interoperability 

Joint  Services,  NATO 

(2) 

IA 

None,  Spoofing,  Hacking,  Flooding 

(4) 

Security 

NIPR,  SIPIR 

(2) 

Message  Type 

Data,  Voice,  Video 

(3) 

Message  Size 

Small,  Medium,  Large,  Mega 

(4) 

Distance  Between  Nodes 

Short,  Average,  Long 

(3) 
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Command  &  Control  Test  Example 

(All  Pairs  Testing  from  ProTest  generates  26  test  cases) 


Factor  A 

Factor  8  Factor  C 

Factor  D  Factor  E 

Factor  F 

Factor  G 

Factor H 

Factor  1 

Factor J 

Factoi K 

Factor 

L  Factor  M 

Factor N 

Factor 0 

Factor  Mission 
Name 

Network  Network 
Size  Load 

Movement 

SAT COM 
Band 

SAT COM 
Angle 

Link 

Degradation 

Node 

Degradation 

EW 

InteroperabilitylA 

Security 

Message 

Type 

Size  of 
Message 

Node 

Distance 

Case  1  Entry 

1 00  nodes  4X 

0TM2 

Combo 

0 

0% 

0% 

None 

NATO 

None 

SIPIFI 

Voice 

Medium 

Short 

Case  2  Consolidation 

10  nodes  Normal 

ATH 

Ka 

45 

5% 

5% 

GPS 

NATO 

Spoofing 

NIPR 

Video 

Large 

Normal 

Case  3  Operation 

50  nodes  2X 

0TM1 

Ku 

75 

20% 

20% 

T  errestrial 

Joint  Serv 

Hacking 

NIPR 

Voice 

Small 

Long 

Case  4  Entry 

50  nodes  2K 

ATH 

Ku 

45 

10% 

10% 

None 

NATO 

Flooding 

NIPR 

Data 

Mega 

Short 

Case  5  Operation 

100  nodes  Normal 

0TM1 

Combo 

75 

10% 

10% 

GPS 

NATO 

Spoofing 

SIPIR 

Data 

Small 

Normal 

Case  G  Operation 

1 0  nodes  4X 

0TM2 

Combo 

45 

0% 

5% 

T  errestrial 

Joint  Serv 

None 

NIPR 

Video 

Mega 

Long 

Case  7  Consolidation 

1 00  nodes  4X 

ATH 

Ka 

75 

20% 

10% 

T  errestrial 

NATO 

Hacking 

SIPIR 

Video 

Medium 

Long 

Case  8  Operation 

10  nodes  Normal 

ATH 

Ka 

o 

20% 

0% 

T  errestrial 

Joint  Serv 

Flooding 

NIPR 

Data 

Large 

Short 

Case  9  (Consolidation 

1 0  nodes  2K 

0TM2 

Ku 

45 

5% 

20% 

None 

Joint  Serv 

Flooding 

SIPIR 

Voice 

Medium 

Normal 

Case  1 0  Consolidation 

50  nodes  2K 

0TM1 

Combo 

o 

0% 

20% 

GPS 

NATO 

None 

NIPR 

Data 

Mega 

Normal 

Case  11  Entry 

50  nodes  Normal 

0TM2 

Ka 

75 

10% 

5% 

GPS 

Joint  Serv 

Hacking 

SIPIR 

Voice 

Large 

Long 

Case  12  Entry 

50  nodes  4X 

0TM1 

Ku 

o 

5% 

0% 

None 

Joint  Serv 

Spoofing 

SIPIR 

Video 

Small 

Long 

Case  1 3  Consolidation 

1 00  nodes  4X 

0TM2 

Ku 

45 

20% 

5% 

GPS 

Joint  Serv 

Flooding 

NIPR 

Data 

Small 

Short 

Case  14  Entry 

1 0  nodes  2< 

DTM1 

Ka 

75 

5% 

0% 

None 

Joint  Serv 

Hacking 

SIPIR 

Data 

Mega 

Normal 

Case  15  Entry 

50  nodes  2X 

ATH 

Ka 

75 

0% 

20% 

T  errestrial 

NATO 

^Spoofing 

NIPR 

Video 

Large 

Short 

Case  1 G  Consolidation 

1 0  nodes  4X 

ATH 

Ku 

o 

10% 

20% 

T  errestrial 

NATO 

None 

NIPR 

Video 

Small 

Normal 

Case  17  Operation 

50  nodes  Normal 

0TM1 

Ku 

75 

0% 

5% 

None 

Joint  Serv 

Flooding 

NIPR 

Data 

Medium 

Short 

Case  18  Operation 

10  nodes  Normal 

0TM1 

Ka 

75 

20% 

10% 

None 

Joint  Serv 

None 

SIPIR 

Video 

Large 

Normal 

Case  19  Operation 

100  nodes  ZK 

0TM2 

Combo 

0 

5% 

10% 

T  errestrial 

NATO 

Hacking 

SIPIR 

Data 

Large 

Short 

Case  20  Consolidation 

100  nodes  Normal 

ATH 

Combo 

P 

20% 

20% 

T  errestrial 

Joint  Serv 

Spoofing 

NIPR 

Voice 

Mega 

Short 

Case  21  (Consolidation 

50  nodes  2X 

0TM1 

Ka 

45 

10% 

0% 

GPS 

Joint  Serv 

Spoofing 

SIPIR 

Data 

Medium 

Normal 

Case  22  Entry 

100  nodes  Normal 

0TM1 

Combo 

o 

20% 

5% 

GPS 

NATO 

Flooding 

NIPR 

Video 

Medium 

Long 

Case  23  Operation 

10  nodes  Normal 

ATH 

Ka 

45 

0% 

10% 

None 

NATO 

Hacking 

SIPIR 

Voice 

Small 

Normal 

Case  24  Entry 

50  nodes  4X 

ATH 

Ku 

45 

5% 

20% 

None 

NATO 

None 

NIPR 

Video 

Large 

Long 

Case  25  Consolidation 

1 0  nodes  2K 

ATH 

Ku 

75 

10% 

5% 

None 

Joint  Serv 

Spoofing 

NIPR 

Data 

Large 

Long 

Case  2G  Consolidation 

100  nodes  Normal 

□TM2 

Combo 

45 

5% 

20% 

GPS 

Joint  Serv 

Spoofing 

NIPR 

Voice 

Mega 

Normal 
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HTT  Applications 


•  Reducing  the  cost  and  time  of  testing  while  maintaining 
adequate  test  coverage 

•  Integration,  interoperability  and  functionality  testing 

•  Creating  a  test  plan  to  stress  a  product  and  discover  problems 

•  Prescreening  before  a  large  DOE  to  ensure  all  2-way 
combinations  are  feasible  before  discovering,  midway  through 
an  experiment,  that  certain  combinations  are  not  feasible 

•  Developing  an  “outer  array”  of  noise  combinations  to  use  in  a 
robust  design  DOE  when  the  number  of  noise  factors  and 
settings  is  large 
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For  More  Information,  Please  Contact 


Mark  Kiemele 

Air  Academy  Associates,  LLC 

1650  Telstar  Drive,  Ste  110 
Colorado  Springs,  CO  80920 


Toll  Free:  (800)  748-1277  or  (719)  531-0777 
Facsimile:  (719)  531-0778 
Email:  aaa@airacad.com 
or  mkiemele@airacad.com 
Website:  www.airacad.com 
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Using  DFSS  as  an  Integrating  Framework 

for  MBT&E  and  DOT&E 


Dr.  Mark  J.  Kiemele 
Air  Academy  Associates 

NDIA  2011  Test  &  Evaluation  Conference 
Tampa,  FL 
14  March  2011 


Mark  J.  Kiemele,  Ph.D. 


u _ 

Office:  719-531-0777 

Cell:  719-337-0357 
mkiemele@airacad.com 
www.airacad.com 

- ^ - 

Using  DFSS  as  an  Integrating  Framework 

for  MBT&E  and  DOT&E 

f 

Dr.  Mark  J.  Kiemele 

Air  Academy  Associates 

NDIA  2011  Test  &  Evaluation  Conference 

Tampa,  FL 

14  March  2011 
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Warm-Up  Exercise 

*  Goal:  full  concentration  on  the  subject 

*  Eliminate  extraneous  issues  that  could  inhibit  that 

*  Write  down  the  top  issue  on  a  plain  sheet  of  paper 

*  Jettison  this  issue  by  doing  the  following: 

-  Design  a  paper  airplane  that  will  help  you  deposit  this 
issue  in  the  waste  basket. 

-  Launch  your  paper  airplane  at  the  waste  basket  from 
your  seating  area.  You  may  stand  or  even  move  around  to 
launch  if  you  wish. 

-  Goal  is  to  put  the  issue  in  the  waste  basket,  which  is 
obviously  symbolic  of  — puMng  the  issue  away.” 
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Food  for  Thought ....  True  or  False? 


The  systems  and  products  that 
deliver  value  to  our  warfighters  are 
perfectly  designed  to  achieve  the 
results  we  are  getting  today. 
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Session  Goals  and  Objectives 


1.  Know  what  DFSS  is  and  understand  that  it  is  a  strategy  that  uses  DOE 
and  other  powerful  methods  to  design,  develop,  and  field  successful 
systems. 

2.  Understand  the  DFSS  process — Identify,  Design,  Optimize,  Validate 
(IDOV) — and  know  that  it  focuses  heavily  on  the  Voice  of  the 
Warfighter. 

3.  Know  that  the  DFSS  process  translates  requirements,  i.e.,  task 
capabilities  and  system  attributes,  into  measures  of  effectiveness  and 
measures  of  performance  and  then  subsequently  into  design 
parameters  which  are  then  optimized  to  produce  highly  capable 
products  and  services. 

4.  Relate  to  some  of  the  powerful  tools  that  are  unique  to  the  DFSS 
process. 

5.  Understand  what  a  transfer  function  is,  be  able  to  comprehend  its 
value,  and  see  that  it  can  be  used  to  develop  linkages  between  Critical 
Operational  Issues  (COIs)  and  measures  of  performance/effectiveness. 

6.  Comprehend  the  opportunity  for  DFSS  in  your  organization  with  regard 
to  MBT&E  and  DOT&E. 
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Agenda 


•  Introduction  and  Review 

•  The  Motivation  for  DFSS 

•  The  DFSS  Process:  Identify,  Design,  Optimize,  Validate  (IDOV) 

-  The  Identify  Phase 

-The  DFSS  Scorecard 
-Voice  of  the  Customer  (VOC) 

-  The  Design  Phase 

-Translating  the  VOC  (Requirements  Flowdown) 

-Concept  Generation  and  Selection 

-Transfer  Functions 

-Critical  Parameter  Management 

-  The  Optimize  Phase 

-Multiple  Response  Optimization 

-Expected  Value  Analysis  Using  Monte  Carlo  Simulation 

-Parameter  Design 

-Tolerance  Allocation 

-  The  Validate  Phase 

-High  Throughput  Testing 

•  Recap  of  DFSS  with  MBT&E  and  Designing  the  Test  and  Evaluation 

•  DFSS  Success  Stories 
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Graphical  Meaning  of  y  and  o 

y  =  Average  =  Mean  =  Balance  Point 
o  =  Standard  Deviation 

i  Point 
-153  =  7 


y  (CTC  performance  measure) 

130  140  150  |  160  170 

y  ~  153 

ct  »  average  distance  of  points 
from  the  centerline 
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99.999943% 

99.9999998% 


Typical  Areas  under  the  Normal  Curve 
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Graphical  View  of  Variation  and 
Performance  Capability 

The  Sigma  rating/capability  of  a  process  performance  measure  is  the  result  of  comparing  the 
Voice  of  the  Process  with  the  Voice  of  the  Customer,  and  it  is  defined  as  follows: 

The  number  of  Sigmas  between  the  center  of  a  process  performance  measured  distribution 
and  the  nearest  specification  limit 

3a  Process  Centered 

•  Process  is  WIDER 
than  the 
specifications, 
causing  waste  and 
cost  of  poor  quality 


6a  Process  Centered 

•  Process  FITS  well 
within  the 
specifications,  so 
even  if  the  process 
shifts,  the  values  fall 
well  within 
tolerances 

-6a  -5a  -4a  -3a  -2a  -la  0  +1a  +2a+3a  +4a+5a+6a 
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Lower 

Specification 

Limit 


3a  Process 


Upper 

Specification 

Limit 


Determined  by 
the  customer 


Determined  by 
the  customer 


-6a  -5a  -4a/  -oa  -2a  -la  0  +1a  +2a  +3aV4a  +5a  +6a 


WASTE 


6a  Process 


WASTE 
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Sigma  Ratings  Measure  Process  Capability 


Sigma  Capability  is  a  measure  of  quality.  It 
compares  the  Voice  of  the  Process  with  the 
Voice  of  the  Customer  and  is  correlated  to  the 
defect  rate.  It  is  computed  from  DPMO. 


Yield  is  the  probability  that  whatever  we 
are  producing  (manufactured  part,  PO, 
shipped  part,  etc.)  will  pass  through  the 
entire  process  without  rework  and 
without  defects. 


a  Capability* 

DPMO* 

RTY 

2 

308,537 

69.1% 

3 

66,807 

93.3% 

4 

6,210 

99.4% 

5 

233 

99.97% 

6 

3.4 

99.99966% 

Process 

Defects  per  Million 

Rolled  Throughput 

Capability 

Opportunities 

Yield 

Academy 

Associates 
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Six  Sigma  is  a  standard  of  Excellence. 

It  means  less  than  4  Defects  per  Million  Opportunities. 

Assumes  a  1.5  sigma  shift  in  average  if  the  performance  measure  is  normally  distributed 
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DFSS:  Getting  to  the  Next  Level 
(the  high  hanging  fruit) 
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Why  DFSS 


"Classic"  Lean  Six  Sigma 
focuses  here 


Research  Design  Development  Production 


Product  Stage 


•  Gain  knowledge  when  costs  are  lowest 

•  Design  in  quality  right  from  the  start 
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DFSS  Goals 

■  Reduce  Cycle  Time  in  the  Design  and 
Development  Process 

■  Reduce  the  Time  to  Money  (TTM) 

■  Reduce  the  Cost  of  Poor  Quality 

■  Improve  Predictability  of  QCD  (Quality,  Cost, 
Delivery) 
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General  Electric  Testimonial 
(Dr.  Norm  Kuchar*,  GECRD,  Oct  2011) 

■  Quality  increases  of  at  least  +1o  at  launch  over 
previous  designs 

- 

■  Time  to  Market  decrease  by  at  least  25%  over 
previous  launches 

f 

■  Cost  savings  due  to  total  resources  utilized  in  the 
20-40%  range 

AIR* 

Academy 

Associates 

Simplify,  Perfect,  Innovate 

*  Norm  was  responsible  for  the  worldwide  deployment  of  GE"s  DFSS  initiative. 
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Launch  Launch 

Early  problem  identification;  solution  when  costs  low 
Faster  market  entry:  earlier  revenue  stream,  longer 
patent  coverage 
Lower  total  development  cost 

Robust  product  at  market  entry:  delighted  customers 
Resources  available  for  next  game-changer 


Pre-DFSS: 
Reactive  Design 

Unhappy  customers  and 
employees 

Unplanned  resource  drain 
Skyrocketing  costs 
Next  product  compromised 


Upfront  investment  is  most  effective  and  efficient 

Show  customers  highly  capable  products  right  from  the  start 
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I 


Reactive 
Design  Quality 


DFSS 


Predictive 
Design  Quality 


From 

To 

•  Evolving  design 

•  Disciplined  requirements 

requirements 

flowdown 

•  Extensive  design 

•  Controlled  design 

rework 

_ ^ 

parameters 

•  Product  performance 

m 

•  Product  performance 

assessed  by  -build 

r 

modeled  and  simulated 

and  test” 

•  Performance  and 

•  Designed  for  robust 

producibility 

performance  and 

problems  fixed  after 

producibility 

product  in  use 

•  Quality  -tested  in” 

•  Quality  -designed  in” 

Lean  Six  Sigma  (DMAIC)  fixes  known  problems. 

DFSS  prevents  unknown  problems  from  occurring. 
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Infamous  Quote 


“As  we  know,  there  are  known  knowns.  These  are 
the  things  we  know  we  know. 

We  also  know  there  are  known  unknowns. 

That  is  to  say  we  know  there  are 
some  things  we  do  not  know. 


But  there  are  also  unknown  unknowns,  the  ones  we 
don"t  know  we  don"t  know.” 
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Donald  Rumsfeld 

Department  of  Defense  news  briefing 
February  12,  2002 
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Identify-Design-Optimize-Validate  (IDOV*) 


i 

Identify 


Define  and  scope  the 
project;  Commission 
team 


X 


Develop  project  plan, 
schedule,  etc. 


X 


Gather  customer 
requirements,  develop 
CTCs  and  prioritize 


X 


Initiate  performance 
scorecard 
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Key  Deliverables: 

•  strategic  plan 

•  benchmarking  results 

•  customer  requirements; 
prioritized,  measurable 
CTCs  (HOQ  #1) 

•  Initial  performance 
scorecard 


Model 


The  IDOV  four-phase  DFSS  process  originated  with  Dr. 
Norm  Kucharat  GE  CRD  and  is  used  with  permission. 


D 

Design 


Determine  specs  for 
all  CTCs 


X 


Develop  alternate 
design  concepts, 
evaluate  risk,  and 
select  best  design 


X 


Translate 
CTCs/functional 
requirements  into 
product  design 
characteristics 


X 


Obtain  transfer 
functions  for  CTCs; 
update  scorecards 
- 1 


Design  concepts  and 
selection  results 
Requirements  flowdown 
Prioritized  product  design 
characteristics  (HOQ  #2) 
Design  risk  assessments 
Transfer  functions 
Updated  scorecards 


Perform  Expected 
Value  Analysis 
(EVA)  for  CTCs 


X 


Perform  Parameter 
Design  (Robust 
Design)  analysis 


X 


Perform  Tolerance 
Allocation  analysis 


X 


Perform  DfX  and 
capability  analysis; 
update  scorecards 


Capability  and 
reliability  studies 
X-ability  assessment 
Optimized  design 
Capability  analysis 
Tolerances  for  key  Xs 
Updated  scorecards 
(capability  flowup) 


Perform  sensitivity 
analysis 


X 


Develop 

Prototypes,  Test 
Cases,  Conduct 
Pilots,  etc. 


X 


Compare  predicted 
and  actual 
capability; 
Complete  gap 
analysis  if  required 


X 


Update  scorecards, 
drawings,  control 
plans,  etc. 


Sensitivity  analysis 
Pilots  /  prototypes  and 
capability  analysis 
Validated  processes  and 
products 

Updated  scorecards 
Control  plan 
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Methods  and  Tools  Used  in  DFSS 

Identify 

Design 

Optimize 

Validate 

Project  or  Study  Charter 

Assign  Specifications 

Histogram 

Sensitivity  Analysis 

Strategic  Plan 

to  CTC"s 

Distributional  Analysis 

Gap  Analysis 

Cross-Functional  Team 

Axiomatic  Design 

Empirical  Data  Distribution 

FMEA 

Voice  of  the  Customer 

Critical  Parameter  Mgt. 

Expected  Value  Analysis  (EVA)  Fault  Tree  Analysis 

Customer  Retention  Grid 

Formulate  Design  Concepts 

Adding  Noise  to  EVA 

Control  Plan 

Benchmarking 

Pugh  Concept  Generation 

Non-Normal  Output  Distributions  PF/CE/CNX/SOP 

KANO"s  Model 

TRIZ 

Design  of  Experiments 

Run/Control  Charts 

Questionnaires 

FMEA 

Multiple  Response  OptimizationM\s\ake  Proofing 

Focus  Groups 

Fault  Tree  Analysis 

Robust  Design  Development  MSA 

Interviews 

Brainstorming 

Using  S-hat  Model 

Reaction  Plan 

Internet  Search 

QFD 

Using  Interaction  Plots 

High  Throughput  Testing 

Historical  Data  Analysis 

Scorecard 

Using  Contour  Plots 

Design  of  Experiments 

Transfer  Function 

Parameter  Design 

Quality  Function  Deployment 

Design  of  Experiments 

Tolerance  Allocation 

Pairwise  Comparison 

Deterministic  Simulators 

Design  For  Manufacturability  and  Assembly 

Analytical  Hierarchy  Process  Discrete  Event  Simulation 

Mistake  Proofing 

Performance  Scorecard 

Confidence  Intervals 

Product  Capability  Prediction 

Flow  Charts 

Hypothesis  Testing 

Part,  Process,  and  SW  Scorecard 

FMEA 

MSA 

Risk  Assessment 

Visualization 

Computer  Aided  Design 

Reliability 

Computer  Aided 

Multidisciplinary  Design  Optimization  (MDO) 

*Unique  to  DFSS 

Engineering 
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Project  Selection:  “DMAIC”  or  -BFSS”? 


•  In  general, 

•  “DMAIC”  approach  and  tools  work  best  when  goal  is  to 
improve  an  existing  product  or  process,  with  baseline 
performance  metrics. 

•  “DFSS”  approach  and  tools  work  best  when  goal  is  to 
design  a  new  product  or  process,  with  no  baseline 
performance  metrics  available,  or  to  redesign  an  existing 
product  or  process  that  is  not  meeting  the  performance 
requirements. 

•  Many  projects  contain  elements  of  both;  use  appropriate  tools, 
without  concern  about  “purity”  of  approach 
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The  DFSS  Process:  Identify,  Design, 

Optimize,  Validate 


-The  Identify  Phase 
-The  DFSS  Scorecard 
-  Voice  of  the  Customer  (VOC) 

-The  Design  Phase 

-Translating  the  VOC  (Requirements  Flowdown) 
-Concept  Generation  and  Selection 
-Transfer  Functions 
-Critical  Parameter  Management 

-  The  Optimize  Phase 

-Multiple  Response  Optimization 

-Expected  Value  Analysis  Using  Monte  Carlo  Simulation 

-Parameter  Design 

-Tolerance  Allocation 

-  The  Validate  Phase 

-High  Throughput  Testing 
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I 


Parts 


Process 


Design 

Scorecard* 


(C 


llln'!0,0( 


'°I0, , 


r- 


t 


Software 


Performance 


*  The  DFSS  Design  scorecard  concept  originated  at  Texas  Instruments  (Defense  Systems  and  Electronics 
Group)  in  the  early  1990s,  and  has  been  adapted  and  used  by  DFSS  practitioners  over  the  past  decade(s). 
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Examples  of  Parts 


Refrigerator 

PARTS  shelves 

drawers 

evaporator 

thermostat 


PROCESS 


f 

V 


weld  sheet  metal 

attach  handle 

attach  handle 

spray  protective 
coating 
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noise  level 
cooling  speed 
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Engraved 

Nameplate 

Statapult® 

metal  plate 

sealant 

pull-back  arm 
pins 

cup 

rubber  band 

align  plate 

attach  protractor 

engrave 

apply  sealant 

attach  cup 

drill  holes 

assemble  side 
panels  to  base 

plate  flatness  ball/cup  fit 
engraving  quality  lateral  dispersion 
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Examples*  of  Scorecard  Entries  for  MBT&E 

Task 

Measure 

Employ  Lethal  Fire  Support 

Time  to  Target  (<  15  min) 

Conduct  Lethal  Direct  Fires 

Positive  Control  Range  (>  50  nm) 
Pssk  (>  .80) 

System  Attribute 

Measure 

Aircraft  TDL 

link  range  (>  60  nm) 

Provide  Lethal  Effects 

Pk/h  (>  .95) 

Guide  Munition 

Ph/s  (>  .90) 

* 

These  examples  are  taken  from  Chris  Wilcox‘s  MBT&E  Tutorial  (page  23)  at  NDIAT&E  2010. 
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Scorecard  Construction 


*  The  scorecard  is  broken  down  into  4  major  areas: 

-  Parts 

-  Process 

-  Performance 

-  Software 

*  A  total  dpu  is  computed  for  each  of  the  four  areas 

*  The  4  dpu‘s  are  summed  to  obtain  a  total  (overall) 
dpu  for  the  entire  product 


Part  DPU 

+ 

Process 

DPU 

+ 

Performance 

DPU 

+ 

Software 

DPU 

Total 
=  DPU 

•  First  Pass  Yield  (FPY)  is  estimated  using  the 
approximation: 

FPY  =  edPu 
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Scorecard  Example  (Nameplate) 


Part  Scorecard 


# 

Part  Name 

DPU 

Qty 

Target 

Continuous  Variable 

Mean  Std  Dev  LSL 

USL 

UOM 

Sample  Size  Known 

Sample  Size  #  Defective 

ppm  Only 

ppm 

1 

plate  thickness 

0.0001083 

1 

0.0625 

0.0614 

0.008  0.03125 

0.09375 

in. 

2 

plate  width 

0.0004306 

1 

1.5 

1.51 

0.015  1.44 

1.56 

in. 

3 

sealant 

0.00005 

1 

50 

Process  Scorecard 


# 

Process  Step 

DPU 

Qty 

Opps 

Continuous  Variable 

Target  Mean  Std  Dev  LSL 

USL  UOM 

Sample  Size  Known 

Sample  Size  #  Defective 

ppm  Only 

ppm 

1 

align  plate  in  fixture 

0.0005000 

1 

1 

500 

2 

engrave 

0.0020000 

1 

1 

1500  3 

3 

apply  sealant 

0.0073333 

1 

1 

1500  11 

Performance  Scorecard 


Continuous  Variable 

Sample  Size  Known 

ppm  Only 

# 

Performance 

DPU 

Qty 

Target  Mean 

Std  Dev  LSL 

USL 

UOM 

Sample  Size 

#  Defective 

ppm 

i 

plate  flatness 

0.0009977 

1 

0.091 

0.011 

0.125 

in. 

2 

engraving  quality 

0.00025 

1 

4000 

1 
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Scorecard  Example  (Nameplate,  cont.) 


Overall  Scorecard  (Roll-Up) 


Scorecard  Summary 

#  Steps/Parts  Total  dpu 

Yield 

dpmo 

ST  Sigma 

LT  Sigma 

Part 

3 

0.000589 

99.94% 

196.31 

5.04 

3.54 

Process 

3 

0.009833 

99.02% 

3,277.78 

4.22 

2.72 

Performance 

2 

0.001248 

99.88% 

623.86 

4.73 

3.23 

Software 

0 

Total 

8 

0.011669983 

98.84% 

1458.748 

4.476 

2.976 
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Pareto  Chart 


Areas  for  Focus: 

Apply  Sealant, 
Engrave, 
Plate  Flatness 


apply  sealant  engrave  plate  flatness  align  plate  in  plate  width  engraving  plate  thickness  sealant 
fixture  quality 

Part,  Process,  or  Performance  Metric 


Statapult  Scorecard  Summary 


Scorecard  Summary 

#  Steps/Parts  Total  dpu 

Yield 

dpmo 

ST  Sigma 

LT  Sigma 

Part 

34 

0.225959 

79.78% 

6,645.86 

3.98 

2.48 

Process 

77 

0.260752 

77.05% 

3,386.39 

4.21 

2.71 

Performance 

4 

0.010783 

98.93% 

2,695.69 

4.28 

2.78 

Software 

0 

Total 

115 

0.497494041 

60.81% 

4,326.035 

4.126 

2.626 

# 

Part  Name 

DPU 

Qty 

Target 

Continuous  Variable 
Mean  Std  Dev 

LSL 

USL  UOM 

Sample  Size  Known 
Sample  Size  #  Defective 

ppm  Only 
ppm 

1 

Base 

2.8666E-07 

1 

0.75 

0.76 

0.012 

0.68 

0.82  inches 

2 

Side  Plates 

0.13137951 

2 

0.75 

0.747 

0.027 

0.7 

0.8  inches 

3 

Cup 

0.05714286 

1 

1  140 

8  1 

4 

Cup  Screw 

0.000014 

1 

14 

5 

Front  Fixed  Arm 

0.00147276 

1 

0.75 

0.745 

0.015 

0.7 

0.8  inches 

6 

Pull  Back  Arm  Length 

9.0705E-05 

1 

14.5 

14.55 

0.12 

14 

15  inches 

7 

Pull  Back  Arm  Width 

0.00095062 

1 

0.75 

0.752 

0.015 

0.7 

0.8  inches 

8 

Angle  Scale 

0.0014 

1 

10000 

14 

9 

Angle  Pointer 

0.00015 

1 

20000 

3 

10 

Removable  Pins 

0.00006 

3 

20 

11 

Nameplate 

0.00025 

1 

250 

12 

Eye  Bolt 

0.0004995 

1 

2002 

1 

13 

Wing  Nut 

0.002 

2 

2000 

2 

14 

Stop  Pad 

0.000031 

1 

31 

15 

Ball 

3.1672E-05 

1 

1_ L5 

1.51 

0.01 

1.45 

1.55  inches| 

16 

Rubber  Band 

0.0001 

1 

100 

17 

Metal  Pins 

0.00039992 

2 

I  10002 

2  1 

18 

Wooden  Peg 

0.00987425 

1 

I  0.375 

0.373 

0.0075 

0.355 

0.395  inchesl 

19 

Wood  Screw 

0.00008 

8 

10 

20 

Plastic  Cap 

0.000032 

2 

16 

21 

Adhesive 

0.02 

1 

|  100 

2  1 

Academy 

Associates 

Simplify,  Perfect,  Innovate 
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Air^ 

Academy 

Associates 


Statapult  Scorecard  Summary  (cont.) 


# 

Process  Step 

DPU 

Qty 

Opps 

Continuous  Variable 
Target  Mean  Std  Dev  LSL 

USL  UOM 

Sample  Size  Known 
Sample  Size  #  Defective 

ppm  Only 
ppm 

1 

drill  cb  through  holes 

0.0039000 

6 

2 

650 

2 

drill  non-cb  through  holes 

0.0072000 

18 

1 

400 

3 

drill  cs  holes 

0.0040000 

1 

2 

2000 

8 

4 

drill  blind  holes 

0.0270000 

9 

2 

1000 

3 

5 

lassemble  fixed  arm  and  side  0.0000000 

1 

6 

1_ L8 

0.045 

5 

NJ 

l  l 

6 

install  wood  screws 

0.1538462 

2 

1 

26 

2 

7 

install  caps 

0.0080000 

2 

1 

1000 

4 

8 

install  wooden  peg 

0.0006667 

1 

1 

3000 

2 

9 

install  angle  scale 

0.0196078 

1 

2 

102 

2 

10 

attach  angle  pointer 

0.0020000 

1 

2 

1000 

2 

11 

attach  rubber  stop  pad 

0.0013316 

1 

2 

1502 

2 

12 

install  cup  on  arm 

0.0010000 

1 

1 

1000 

13 

install  removable  pins 

0.0002000 

2 

1 

10002 

1 

14 

| insert  arm  between  side  plat<  0.0020000 

1 

2 

1000 

2 

15 

assemble  rubber  band 

0.0200000 

1 

3 

150 

3 

16 

attach  name  plate 

0.0100000 

1 

2 

100 

1 
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# 

Performance 

DPU 

Qty 

Target 

Continuous  Variable 
Mean  Std  Dev 

LSL 

USL 

UOM 

Sample  Size  Known 
Sample  Size  #  Defective 

ppm  Only 
ppm 

1 

gap 

0.006252391 

1 

0.06 

0.014 

0.005 

0.095 

inches 

2 

distance 

0.003830381 

1 

162 

4.5 

150 

inches 

3 

life 

0.0002 

1 

200 

4 

wood  grain  quality 

0.0005 

1 

4000  2 

Simplify,  Perfect,  Innovate 


\ 


Academy 

Associates 


Understanding  the  Voice-of-the-Customer 

(VOC) 


Suddenly,  a  heated  exchange  took  place 
between  the  king  and  the  moat  contractor. 

Source:  The  Far  Side 
The  Far  Side  Millennium  Off-the-Wall  Calendar  2000 

Far  Works,  Inc. 
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Simplify,  Perfect,  Innovate 


Quality  Function  Deployment  (QFD) 


Task 

Capabilities 
and  System 
Attributes 


Measures  of 
Effect./Perf. 


/ 


Critical  to  Customer  (CTC)  Performance  Measures 


In  DFSS 
terms,  these 
are  the  VOC 
(Voice  of  the 
Customer) 


cr 

0 

01 

~o 

0 

N 


o 

o_ 


House  of 
Quality 
#1  ' 


Prod.  Design 
Characteristics 


Prioritized 
MOEs  +  MOPs 


■0  0 
0  0 
N  h= 

House  of 

:=  1 
■g  3> 

CL  ^ 

Quality 

#2 

Mfg.  Process 
Characteristics 


Prioritized 
Design  Char"s 


Prior.  Mfg. 
Process  Char's 


Academy 

Associates 


Marketing 

•  Features 

•  Quality 

•  Performance 

•  Cost 


Design  Engineering 

•  Performance 

•  Reliability 

•  Cost 


Mfg.  Engineering 

•  Manufacturability 

•  Cost 


Mfg.  Process 
Control 


Mfg.  Process 
Control 


Manufacturing 

•  SPC 

•  Process  Capability 
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Simplify,  Perfect,  Innovate 


Academy 

Associates 


Voice  of  the  Customer 
(Refrigerator) 


voc 


— Writ  it  to  be  energy  efficient” 

— Wnt  it  to  be  quiet” 

— Neds  to  preserve  food” 

— Wnt  to  be  able  to  easily 
reconfigure  the  shelves” 

— Wnt  to  fit  large,  bulky  items” 

— Sbuld  last  a  long  time” 

— Wuld  like  it  to  match  my 
kitchen” 


i—  Operational 


energy 

efficient 


quiet 


preserves 

food 


Affinity  Diagrams 
Can  Help 


i—  Usability 


easy  to 
reconfigure 


stores 
large,  bulky 
items 


i—  Reliability  — i  r~  Installation  -i 


lasts  a  long 
time 


low  service 
costs 


matches 

kitchen 


fits  in 
allocated 
space 
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Simplify,  Perfect,  Innovate 


Academy 

Associates 

Simplify,  Perfect,  Innovate 


Use  Pairwise  Comparison  to  Prioritize 


Customer  Requirements' 

A 

B 

C 

D 

E 

F 

G 

Total 

A 

energy  efficient 

X 

X 

X 

X 

X 

X 

X 

1 

B 

quiet 

B 

X 

X 

X 

X 

X 

X 

4 

C 

preserves  food 

C 

C 

X 

X 

X 

X 

X 

5 

D 

easy  to  reconfigure 

D 

D 

D 

X 

X 

X 

X 

6 

E 

handles  large,  bulky  items 

E 

B 

C 

D 

X 

X 

X 

2 

F 

lasts  a  long  time 

F 

B 

C 

D 

F 

X 

X 

3 

G 

matches  kitchen 

A 

B 

C 

D 

E 

F 

X 

0 

Example:  -Easy 
to  reconfigure” 
won  6  times. 


Used  to  set 
weights  for 
HOQ1 
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Academy 

Associates 

Simplify,  Perfect,  Innovate 


Place  Customer  Requirements  &  Rating  into 
HOQ  #1  (Refrigerator  Example) 


Grouped 


Performance  Measures 


customer  Rating 

requirements  \ 

i  \ 

y 

A:  energy  efficient 

2 

B:  quiet 

4 

C:  preserves  food 

5 

D:  easy  to  reconfigure 

5 

E:  handles  large,  bulky  items 

3 

F:  lasts  a  long  time 

4 

G:  matches  kitchen 

1 

Rating: 


5 

4 

3 

2 

1 


Must  have  for  performance 
Highly  desirable  feature 
Desirable  feature 
Usable  feature  but  not  critical 
Nice  feature  but  not  critical 
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Academy 

Associates 


Simplify,  Perfect,  Innovate 


Fill  in  Performance  Measures  Across  Top 


Peformance  Measures 
(CTCs) - * 

energy  efficiency 

rating 

noise  level  (db) 

temperature  range 

cooling  speed 

(sec.  per  degree) 

%  adjustable 

shelves 

> 

</> 

(/>  ^ 
CO  c 

0)  5 

>»  <1 
(0  c 

£  .s 

cu  +- 

shelf  depth  and 

width  (in.) 

door  tray  depth  (in.) 

mean  time  to 

failure  (hrs) 

#  available  colors 

A:  energy  efficient 

2 

B:  quiet 

4 

C:  preserves  food 

5 

D:  easy  to  reconfigure 

5 

E:  handles  large,  bulky 
items 

3 

F:  lasts  a  long  time 

4 

G:  matches  kitchen 

1 
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Relationships 

■  Now,  determine  the  strength  of  the  relationships 
between  the  customer  requirements  and  the  CTCs. 
Rate  the  relationship  between  each  customer 
requirement  and  each  CTC  according  to  the  scale 
below. 

9:  Strong  Relationship 

3:  Medium  Relationship 

f 

1 :  Weak  Relationship 

Blank:  No  Relationship 

■  Compute  a  Rank-Ordered  Sum  for  each  CTC 

AIR* 

Academy 

Associates 

Simplify,  Perfect,  Innovate 

(multiply  strength  •  rating  and  add) 
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f 


Academy 

Associates 


HOQ  #1  ....  Prioritizes  the  Performance  Measures 


energy  efficiency  rating 

noise  level  (db) 

temperature  range 

cooling  speed 

%  adjustable  shelves 

disassy  /  reassy  time  (sec.) 

shelf  depth  and  width  (in.) 

door  tray  depth  (in.) 

mean  time  to  failure 

#  available  colors 

CTCs/FPs  1  ^ 

(Functional  Domain) 

voc 

(Customer  Domain) 

Customer  Requirements: 

Importance 

Rating 

A:  energy  efficient 

2 

9 

1 

3 

9 

1 

1 

1 

B:  quiet 

4 

3 

9 

1 

3 

C:  preserves  food 

5 

3 

9 

9 

1 

1 

1 

D:  easy  to  reconfigure 

5 

3 

9 

E:  handles  large,  bulky  items 

3 

9 

1 

9 

9 

F:  lasts  a  long  time 

4 

1 

1 

9 

G:  matches  kitchen 

1 

9 

c 

V. 

Weigh 

ted  Sums  »> 

49 

38 

55 

79 

42 

48 

34 

34 

43 

9 

Task  Capabilities  and  System 
Attributes  are  mapped  to 
Performance  Measures 
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Simplify,  Perfect,  Innovate 


Prioritized  Measures  Become  Side  of  HOQ  #  2 


Task 

Capabilities 
and  System 
Attributes 


Measures  of 
Effect./Perf. 

Prioritized  Req. 

House  of 
Quality 
#1  ' 

Factors/Conditions  that  Affect  the 
Performance  Measures  (MBT&E) 


Prod.  Design 
Characteristics 


Prioritized 
MOEs  +  MOPs 


Mfg.  Process 
Characteristics 


Prioritized 
Design  Char"s 


t 

Prioritized 

Factors/Conditions  (MBT&E) 


Prior.  Mfg. 
Process  Char's 


Academy 

Associates 


Marketing 

•  Features 

•  Quality 

•  Performance 

•  Cost 


Design  Engineering 

•  Performance 

•  Reliability 

•  Cost 


Mfg.  Engineering 

•  Manufacturability 

•  Cost 


Mfg.  Process 
Control 


Mfg.  Process 
Control 


Manufacturing 

•  SPC 

•  Process  Capability 
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Simplify,  Perfect,  Innovate 


AU^ 

Academv 

Associates 


Simplify,  Perfect,  Innovate 


Key  Features  (VOC) 

Self-heating 
Activated  by  button 


on  bottom  of  can 


Used  for  hot 
beverages  and 
soups 

Disposable 

Environmentally 

compatible 
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Customer  Appeal 


CONCEPT 


PRODUCT 


(Packaging) 


(Taste) 


■  —Potable” 

■  — Stf  Heating 

■  — Eay  to  Use 

■  — Uniqe” 


jj 


jj 


— tot  bitter” 
■  —  Frefe” 
—Smooth 
— tfrieties” 


Simply. .. 


Academv 

Associates 


I  want  to  be  able  to  buy  one  cup  with  a  button  on  it, 
press  the  button,  and  it  gets  hot,  and  tastes  great, 
then  I  can  throw  it  away  when  I’m  done. 


Simplify,  Perfect,  Innovate 
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Academy 

Associates 

Simplify,  Perfect,  Innovate 


voc 

Prioritization 


A 

B 

C 

D 

E 

F 

G 

H 


Portable 


Self  Heating 


Easy  to  Use 


Unique 


Not  bitter 


Fresh 


Smooth 


Varieties 


0) 

n 

<0 

tr 

o 

Cc 


A 


U) 

c 


(U 

0 


0 

CO 


B 


B 


© 

</> 


</> 

<0 

LU 


D 

E 

F 

G 

L. 

© 

0 

3 

■ 

o 

(/> 

o 

C 

O 

0 

E 

3 

Z 

LL 

CO 

0) 

0 


0 

U 

rc 


Example: 
“Self-Heating” 
won  5  times. 


Pareto  of  Pairwise  Comparion  Results 


Used  to  set 
weights  for  HOQ1 


Self  Easy  to  Use  Not  bitter  Portable  Fresh  Unique  Smooth  Varieties 
Heating 

VOC  ITEM 
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VOC  CTCs 


Customer  Domain  Language  to 
Functional  Domain  Language 
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CTCs/Perf.  Measures 

(Functional  Domain) 


voc 

(Customer  Domain) 

wt 


Jl/ *9  o 


Portable 

3 

9 

3 

9 

3 

3 

Self  Heating 

5 

3 

9 

9 

9 

3 

Easy  to  Use 

4 

3 

1 

3 

3 

Unique 

2 

■ 

voice  oi  me  ousiumer  is 
mapped  to  Functional  or  ) 

■ 

■ 

Performance  Measures 

Scores  are  multiplied  by  weights  and  summed  below 


Academy 

Associates 

Simplify,  Perfect,  Innovate 


Prioritized  CTCs 


54 

58 

72 

66 

36 

. 

. 
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The  Design  Phase 


Academy 

Associates 
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Simplify,  Perfect,  Innovate 


Academy 

Associates 


Simplify,  Perfect,  Innovate 


The  DFSS  Process:  Identify,  Design,  Optimize,  Validate 


-  The  Identify  Phase 

-The  DFSS  Scorecard 
-Voice  of  the  Customer  (VOC) 

-The  Design  Phase 

-Translating  the  VOC  (Requirements  Flowdown) 
-Concept  Generation  and  Selection 
-Transfer  Functions 
-Critical  Parameter  Management 

-  The  Optimize  Phase 

-Multiple  Response  Optimization 

-Expected  Value  Analysis  Using  Monte  Carlo  Simulation 

-Parameter  Design 

-Tolerance  Allocation 

-  The  Validate  Phase 

-High  Throughput  Testing 
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Systems  Engineering 


Main  System 


Sub  System 


Assemblies 


Parts 


Automobile 


ll 


Complex  products  may  require  the  "Divide  and  Conquer 
approach. 


*  Requirements  are  flowed  down,  while  capabilities  are  rolled  up. 

*  System  Engineers  are  the  masters  of  the  scorecard  and  make 
tradeoff  decisions. 
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Requirements  Flowdown  Using  QFD 


Functional  Req. 
(CTC's! 


Performance  Measures 


Customer 

Expectations 


cr 

0 

House  of 

0 

E 

o 

Quality 

0 

3 

#1  ' 

O 

* 

Prod.  Design  | 
Characteristics  1 

& 

unctional  Re 

(CTC's) 

House  of 
Quality 
#2 

LL 

Mfg.  Process 
Characteristics 


* 


Product 
Design  CTC's 


Academy 

Associates 


Marketing 

•  Features 

•  Quality 

•  Performance 

•  Cost 


Design  Engineering 

•  Performance 

•  Reliability 

•  Cost 


Mfg.  Engineering 

•  Manufacturability 

•  Cost 


Mfg.  Process 
Control 

Mfg.  Process 

Characteristics 

House  of 
Quality 
#4 

J 

n 

L 

Mfg.  Process 
Control 


Manufacturing 

•  SPC 

•  Process  Capability 
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Simplify,  Perfect,  Innovate 


Formulate  Design  Concepts 

•  Create  alternative  designs  that  fulfill  CTC'ls. 

•  Compare  designs  with  functional  requirements  (CTC'ls) 

•  Choose  the  best  design 

-  How  do  we  decide  which  is  the  best  approach? 

f 

•  Assess  risk  of  chosen  design. 

\ 

•  Tools  for  Concept  Generation  and  Selection 

•  Axiomatic  Design 

•  TRIZ 

AIR* 

Academy 

Associates 

Simplify,  Perfect,  Innovate 

•  Pugh  Concept  Selection 

©2011  Air  Academy  Associates,  LLC.  Do  Not  Reproduce.  Page  56 

Case  Study:  General  Design  Concept 


Beverage 


Convection 


Energy  release 
Calcium  Oxide  (CaO) 


Water  for  reaction 


Point  of  activation 
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Academy 

Associates 


FAST  allows  us  to  quickly  design  the  key 
functionality  of  the  product  (system). 


GENERATE 

HEAT 


CAUSE  A 
REACTION 


FLOOD 

CALCIUM 

OXIDE 


BREAK 

WATER 

CELL 


PRESS 

BUTTON 


HOW? 


WHY? 


All  of  the  “hows”  and  “whys”  must  be  answered  (both 
directions). 
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Simplify,  Perfect,  Innovate 


Air^ 

Academy 

Associates 


DPs 

(Physical  Domain) 


CTCs  (FPs) 

(Functional  Domain) 


Can  Temp 
Time  to  Use 


Scores  are  multiplied  by  weights  and  summed  below 


1  1 

Prioritized  DPs 

57 

75 

67 

- 

- 

- 
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Simplify,  Perfect,  Innovate 


Transfer  Function:  The  Bridge  to  Innovation 


r 


Parameters 
or  Factors 
that 

Influence 
the  CTC 


v 


S  =f2(x1,x2,  x3) 


Academy 

Associates 

Simplify,  Perfect,  Innovate 


Where  does  the  transfer  function  come  from? 

•  Exact  transfer  function 

•  Approximations 

-  DOE 

-  Historical  Data  Analysis 

-  Simulation 
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Academy 

Associates 

Simplify,  Perfect,  Innovate 


Exact  Transfer  Functions 


Engineering  Relationships 
-  V=  IR 


F  =  ma 


9V  -r 


X 


r 


£  < 


v 


> 

> 

> 

> 

> 

> 

Where 


N 

I 

r 

t 

x 

H 


The  equation  for  current  (I)  through 
this  DC  circuit  is  defined  by: 


/= 


v  \ m+Rz) 


fl.- ff2  R,R2 


r,+r2 


The  equation  for  magnetic  force  at  a  distance 
X  from  the  center  of  a  solenoid  is: 


H  =  — 

21 


.5£  +  x 


+ 


.5£  -x 


^r2  +  (.5^  +  x)2  ^r2  +(.5^-x) 


total  number  of  turns  of  wire  in  the  solenoid 
current  in  the  wire,  in  amperes 
radius  of  helix  (solenoid),  in  cm 
length  of  the  helix  (solenoid),  in  cm 
distance  from  center  of  helix  (solenoid),  in  cm 
magnetizing  force,  in  amperes  per  centimeter 
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Hierarchical  Transfer  Functions 


Y  =  Gross  Margin  = 


Gross  Profit 


Gross  Revenue 


Y  =  f(yi,  y2,  y3,  y4.  y5.  y6) 

y i  y2  y3  /  y4  \  y5  Ye 

"  (R©VeqUjp  -  COG)  +  (ReVpost  sales'T  salcs)f  ( RG Vfjn  COStfjn) 

Yi  +  y3  +  Ys 


Academy 

Associates 


y4 

Costpost  saies  =  f(field  cost,  remote  services,  suppliers) 


x1  =  f(direct  labor,  freight,  parts,  depreciation) 
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Simplify,  Perfect,  Innovate 


What  is  a  Designed  Experiment? 


Purposeful  changes  of  the  inputs  (factors)  in  order  to  observe 
corresponding  changes  in  the  output  (response). 


x, 


x. 


Inputs  x. 


x. 


* 

* 


PROCESS 


Yi 


Outputs 


Academy 

Associates 


Run 

X,  X2  X3  X4 

CM 

>- 

T— 

>- 

Y 

Sy 

1 

2 

3 
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Simplify,  Perfect,  Innovate 


DOE  Helps  Determine  How  Inputs  Affect 


Outputs 

i) 

Factor  A  affects  the  average  of  y 

y 

ii) 

Factor  B  affects  the  standard  deviation  of  y 

ABi 

y 

iii) 

Factor  C  affects  the  average  and  the 

f 

standard  deviation  of  y 

iv) 

Mo, 

/  \  1 

y 

Air^ 

Factor  D  has  no  effect  on  y 

Academy 

Associates 

=  D2 

Simplify,  Perfect,  Innovate 

y 
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Catapulting  Power  into  DFSS 


Statapult®  Catapult 
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The  Theoretical  Approach  (cont.) 


Io0  =  rFF(0)sin0coscp-(MgrG  +  mgrB)sin0 

0 


tan<|)  = 


D-rF  sin0 
d  +  rF  cos0’ 


1 


lo0  =  rFl  F(0)sin0  coscpd0-(MgrG  +mgrB)(sin0-sin0o) 


1 


% 


lo02  =  rF  F(0)sin0coscpd0-(MgrG  +mgrB)(sin01  -sin0o). 


x  =  vB  cos 


^TT  n\  1 


-01  t  —  rBcos01 


j 


y  =  rBsin01+vBsin  --01  t--gt2 

J  *- 


1 


Academy 

Associates 


^71  ^ 


rB  sin0-| +(R  +  rB  cos0-|)tan  — 0-| 


g  (R  +  rBcos  ©0  _0 

2  „  'N  _U- 


^  J  2V|cos2 


vf-°l 


gio 


(R  +  rBcos0-|  Y 


4rB  2 
cos 


(n  0  1 
2“01 

Vz  J 


f  n  ^ 


rB  sin01 +(R  +  rB  cos0-|)tan  — 0-| 

v2  j 


9| 


^F{F(e)sinecos*de-(MgrG  +  mgrB)(sine1-sine0). 

% 
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Simplify,  Perfect,  Innovate 


Statapult®  DOE  Demo 

(The  Empirical  Approach) 


Academy 
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Actual 

Factors 


Coded  Factors 


Response  Values 
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Simplify,  Perfect,  Innovate 


What  Makes  DOE  so  Powerful? 
(Orthogonality:  both  vertical  and  horizontal  balance) 

A  Full  Factorial  Design  for  3  Factors  A,  B,  and  C,  Each  at  2  levels: 

Run 

A 

B  C 

AB 

AC  BC 

ABC 

1 

- 

- 

+ 

+  + 

- 

2 

- 

+ 

+ 

- 

+ 

3 

- 

+ 

- 

+ 

+ 

f 

4 

- 

+  + 

- 

+ 

- 

5 

+ 

- 

- 

+ 

+ 

6 

+ 

+ 

- 

+ 

- 

AIR* 

7 

+ 

+ 

+ 

Academy 
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8 

+ 

+  + 

+ 

+  + 

+ 
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Value  Delivery:  Reducing  Time  to  Market  for 

New  Technologies 


INPUT 


Pitch  <) 

(0,  15,  30) 

Roll  <) 

(0,  15,  30)  . 

W1F<) 

(-15, 

0,  15) 

W2F  <) 

(-15, 

0,  15) 

W3F  <) 

(-15, 

0,  15)  . 

Modeling  Flight 
Characteristics 
of  New  3-Wing 
Aircraft 


OUTPUT 


Six  Aero- 


Characteristics 


Total  #  of  Combinations  =  35 
Central  Composite  Design:  n 


243 

30 


Patent  Holder:  Dr.  Bert  Silich 
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Aircraft  Equations 


II 

_i 

o 

.233  +  .008(P)2  +  .255(P)  +  .012(R)  -  .043(WD1)  -  .117(WD2)  +  .185(WD3)  +  .010(P)(WD3)  - 
.042(R)(WD1)  +  .035(R)(WD2)  +  .016(R)(WD3)  +  .010(P)(R)  -  .003(WD1)(WD2)  - 
.006(WD1)(WD3) 

o 

D 

II 

.058  +  .016(P)2  +  .028(P)  -  .004(WD1)  -  .013(WD2)  +  .013(WD3)  +  .002(P)(R)  -  .004(P)(WD1) 

-  .009(P)(WD2)  +  .016(P)(WD3)  -  .004(R)(WD1)  +  .003(R)(WD2)  +  .020(WD1)2  +  .017(WD2)2 
+  .021  (WD3)2 

II 

>- 

o 

-.006(P)  -  .006(R)  +  .169(WD1)  -  .121(WD2)  -  .063(WD3)  -  .004(P)(R)  +  .008(P)(WD1)  - 
.006(P)(WD2)  -  .008(P)(WD3)  -  .012(R)(WD1)  -  .029(R)(WD2)  +  .048(R)(WD3)  -  .008(WD1)2 

f 

Cm  = 

.023  -  .008(P)2  +  .004(P)  -  .007(R)  +  .024(WD1)  +  .066(WD2)  -  .099(WD3)  -  .006(P)(R)  + 
.002(P)(WD2)  -  .005(P)(WD3)  +  .023(R)(WD1)  -  .019(R)(WD2)  -  .007(R)(WD3)  +  .007(WD1)2 
-  .008(WD2)2  +  .002(WD1)(WD2)  +  .002(WD1)(WD3) 

j\ 

II 

>- 

o 

.001  (P)  +  .001  (R)  -  .050(WD1)  +  .029(WD2)  +  .012(WD3)  +  .001(P)(R)  -  .005(P)(WD1)  - 
.004(P)(WD2)  -  .004(P)(WD3)  +  .003(R)(WD1)  +  .008(R)(WD2)  -  .013(R)(WD3)  +  .004(WD1)2 
+  .003(WD2)2  -  .005(WD3)2 

AIR* 

Academy 

Associates 

Simplify,  Perfect,  Innovate 

ce  = 

.003(P)  +  .035(WD1)  +  .048(WD2)  +  .051  (WD3)  -  .003(R)(WD3)  +  .003(P)(R)  -  .005(P)(WD1) 

+  .005(P)(WD2)  +  .006(P)(WD3)  +  .002(R)(WD1) 
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Fusing  Titanium  and  Cobalt-Chrome 


Courtesy  Rai  Chowdhary 
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DOE- 

— Maket  Research” 

Example 

Suppose  that,  in  the  auto  industry,  we  would  like  to  investigate  the  following  automobile  attributes 

(i.e.,  factors),  along  with  accompanying  levels  of  those  attributes: 

A:  Brand  of  Auto: 

-1  =  foreign 

+1 

=  domestic 

B:  Auto  Color: 

-1  =  light 

0  =  bright 

+1 

=  dark 

C:Body  Style: 

-1  =  2-door 

0  =  4-door 

+1 

=  sliding  door/hatchback 

D: Drive  Mechanism: 

-1  =  rear  wheel 

0  =  front  wheel 

+1 

=  4-wheel 

E:  Engine  Size: 

-1  =  4-cylinder 

0  =  6-cylinder 

+1 

=  8-cylinder 

F:  Interior  Size: 

-1  <  2  people 

0  =  3-5  people 

+1  >  6  people 

G:  Gas  Mileage: 

-1  <  20  mpg 

0  =  20-30  mpg 

+1  >  30  mpg 

H:Price: 

-1  <  $20K 

0  =  $20-$40K 

+1  >  $40 K 

In  addition,  suppose  the  respondents  chosen  to  provide  their  preferences  to  product 

profiles  are  taken  based  on  the  following  demographic: 

J:  Age: 

-1  <  25  years  old 

+1  >  35  years  old 

K:  Income: 

-1  <  $30K 

+1  >  $40 K 

L:  Education: 

-1  <  BS 

+1  >  BS 

©2011  Air  Academy  Associates,  LLC. 
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Simplify,  Perfect,  Innovate 


DOE  — Maket  Research”  Example  (cont.) 


Question:  Choose  the  best  design  for  evaluating  this  scenario 


Answer: 


L18  design  with  attributes  A  -  H  in  the  inner  array  and 
factors  J,  K,  and  L  in  the  outer  array,  resembling  an 
L18  robust  design,  as  shown  below: 


Academy 

Associates 

Simplify,  Perfect,  Innovate 


Run* 


1 

2 

3 

4 

5 

6 

7 

8 

9 

10 
11 
12 

13 

14 

15 

16 

17 

18 


A  B  C  D 


F  G  H 


o 

+ 

0 

+ 

0 

+ 


0 

0 

+ 


L 

K 

J 


18  different  product  profiles 


+  -  + 
+  + 


+ 

+  + 

+  + 


yi  y2  y3  y4  y5  y6  y7  y8 


Segmentation  of  the  population  or 
Respondent  Profiles 
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Modeling  The  Drivers  of  Turnover* 
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Simplify,  Perfect,  Innovate 


© 


External  Market  Factors 
(Local  Labor  Market  Conditions) 


Local  Unemployment  Rate 


Local  Employment  Alternatives 


Company"s  Market  Share 


Organizational  Characteristics 
and  Practices 


Supervisor  Stability 


Lateral  /  Upward  Mobility 


Layoff  Climate 


(D  Employee  Attributes 


Time  Since  Last  Promotion 


Education  Level 


Job  Stability  History 


Process  of 
Deciding  to 
Stay  /  Leave 


Turnover  Rate 


*  Adapted  from  Harvard  Business  Review  article  on  Boston  Fleet  Bank,  April  2004,  pp  116-125 
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Simplify,  Perfect,  Innovate 


The  Value  of  Transfer  Functions 


■  Provide  a  simple  and  compact  wav  of  understanding 
relationships  between  performance  measures  or  response 
variables  (y‘s)  and  the  factors  (x‘s)  that  influence  them. 

■  Allow  for  the  prediction  of  the  response  variable  (y),  with 
associated  risk  levels,  before  any  change  in  the  product  or 
process  is  made. 

■  Allow  for  the  assessment  of  process  or  product  capability  in 
the  presence  of  uncontrolled  variation  or  noise. 

■  Allow  the  very  quick  manipulation  of  complex  systems  using 
Monte  Carlo  Simulation  (i.e.,  Expected  Value  Analysis)  for  the 
purpose  of  assessing  risk. 

■  Provide  a  very  easy  wav  to  optimize  performance  via  robust  or 
parameter  design  and  tolerance  allocation. 

■  Make  sensitivity  analysis  easy  and  straightforward. 

■  Greatly  enhance  one‘s  knowledge  of  a  product  or  process. 

■  In  general,  they  are  the  gateway  to  systematic  innovation. 

■  Provide  a  meaningful  metric  for  the  maturity  in  DFSS  for  any 
organization. 
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Case  Study:  Transfer  Functions 


Example:  — Tme  to  use”  and  — Qn  temp”  as  a  function  of 
— WII  thickness”,  “CaO  mass”,  and  — yo  volume” 


Wall  thickness  (XI) — H 

CaO  mass  (X2) - h 

H20  volume  (X3) - h 


Y1=f1(X1,X2,  X3) 
Y2=f2(X1 ,  X2,  X3) 


■>  Time  to  use  (Y1) 
-►  Can  temp  (Y2) 


How  do  we  find  the  functions  f1  and  f2? 

•  First  principle  equations 

(Physics  /  Engineering  equations) 

•  Analytical  Models  (Simulation  and  Regression) 

FEA,  CFD,  etc. 

•  Empirical  models  (Design  of  Experiments) 
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Factor 
Row  # 

1 

A 

Wall  Thick 
-1 

B 

C 

Time  to  use 

CaO  Mass 

H20  Volume 

Y1 

Y2  Y3  Y4  Y5 

-1 

-1 

15.41278 

15.16945 

13.12958 

16.572  15.81977  10.29423  10.92083 
16.6466  14.76726  9.096769  12.39497 
16.93513  17.3345  13.11604  11.0545 

2 

-1 

-1 

1 

3 

-1 

1 

-1 

4 

Factor 

A 

B 

C 

Max  can  temp 

5 

Row  # 

Wall  Thick 

CaO  Mass 

H20  Volume 

Y1  Y2  Y3  Y4 

Y5 

6 

1 

-1 

-1 

-1 

108.1381  103.948  107.5071  102.9392  104.3727 

7 

2 

-1 

-1 

1 

109.8235  105.5967  108.1956  109.2639  104.0185 

8 

3 

-1 

1 

-1 

108.0081  109.2598  110.8279  109.7068  109.4276 

Y-hat  Contour  Plot  of  (Time  to  use)  Wall  Thickness  vs  CaO  Mass 


niMB 

ir-u 

IVlb 

14.15 

15-14 

IMS 

AIM.' 

tan 

■  9-10 

mw 

Of -6 
16.7 

iW 

■  4-5 

□  3-4 
o2  3 


10-09-06417  06-0 


03  0301  00  01  0?  03 

Wall  ThkiiiM* 


Y-hat  Model 

Time  to  use 

Max  can  temp 

1 

■a* 

> 

> 

Factor 

Name 

Coeff 

P(2  Tail) 

Tol 

< 

Coeff 

P(2  Tail) 

Tel 

'5 

< 

Const 

17.055 

0.0000 

106.91 

0.0000 

A 

Wall  Thickness 

0.73753 

0.0444 

1 

X 

0.71745 

0.0302 

1 

X 

B 

CaO  Mass 

-0.17772 

0.6237 

1 

X 

0.56822 

0.0842 

1 

X 

C 

H20  Volume 

0.49940 

0.1282 

1 

X 

AB 

-0.91022 

0.0269 

1 

X 

-1.564 

0.0001 

1 

X 

AC 

-1.128 

0.0027 

1 

X 

_ _ 

_1  pqq 

0.0068 

1 

X 

2.478 

0.0000 

1 

X 

Y-hM  Surface  Plot  or  {Time  to  use]  Wall  Thickness  vs  CaO  Mass 


I  13 


l?-ia 
ii5-i7 
it- It, 
■  14-15 
flljj-H 
£17-13 
1 1  i-12 
P 10-11 
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Inc:  37 

Time:  7.565e-001 
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Simplify,  Perfect,  Innovate 


Computational  Fluid 
Dynamics  (CFD)  for 
modeling  temperature 
(fluid)  components 


MSC)< 


Inc:  40 

Time:  3.300e+003 


Finite  Element  Analysis 
(FEA)  for  modeling  structural 
components 


Temperature  (Integration  Point) 
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FEA/CFD  Model 


r 
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Predicted  results 
validated  in  model 


Regression  Modeling 


Y-hat  Model 

Factor 

Name 

Response  #1 

Coeff 

P(2  Tail) 

Active 

Response  #2 

Coeff 

P(2  Tail) 

H 

Active 

Response  #3 

Coeff 

P(2  Tail) 

Active 

Const 

-0.45862 

0.9726 

10.817 

0.3815 

-8.245 

0.4975 

Row  # 

A 

B 

c 

1 

-1 

-1 

-1 

2 

-1 

-1 

1 

3 

-1 

1 

-1 

4 

-1 

1 

1 

5 

1 

-1 

-1 

6 

1 

-1 

1 

7 

1 

1 

-1 

8 

1 

1 

1 

9 

0 

0 

0 

10 

0 

0 

0 

11 

-1 

0 

0 

12 

1 

0 

0 

13 

0 

-1 

0 

14 

0 

1 

0 

15 

0 

0 

-1 

Response  #1 


Y1 


Y2 


o 


-12 

-63 

-47 

-81 

71 

62 

-55 

-94 

96 

-100 

25 

45 

78 

-18 

-90 

-53 


-79 

87 

57 

74 

-89 

49 

-67 

-62 

-66 

11 

-16 

44 

-50 

-54 

-30 

18 


4 

6 

32 

70 

-31 

-37 

95 

54 

65 

95 

-83 

23 

-31 

-3 

85 

-94 


-1 

83 

38 

-54 

-50 

-68 

25 

-95 

22 

-65 

74 

-8 

91 

-43 

48 

-26 


Total  |  290055.5_ 79 


271616.0  79 


241854.9_ 79 


Prediction 


Factor 

Name 

Low 

High 

Exper 

A 

Wall  Thickness 

-1 

1 

-1 

B 

CaO  Mass 

-1 

1 

-1 

C 

H20  Volume 

-1 

1 

-0.943791176 

Multiple  Response  Prediction 

99%  Confidence  Interval 

Y-hat 

S-hat 

Lowei  Bound 

Upper  Bound 

Time  to  use 

13.9460 

0.6050 

12.131 

15.761 

Max  can  temp 

105.0000 

1.1425 

101.573 

108.428 
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18 

-63 

68 

11 

79 

74 

31 

43 

41 

-70 

-23 

58 

-69 

-75 

73 

61 
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Critical  Parameter  Management  and  COIs 


-A Critical  Operational  Issue  (COI)  is  linked  to  operational  effectiveness  and 
suitability. 

-  It  is  typically  phrased  as  a  question,  e.g., 

Will  the  system  detect  the  threat  in  a  combat  environment  at 
adequate  range  to  allow  for  successful  engagement ? 


y2  (engagement) 


xl  x2  x3  (ranges)  x4  (threat  type)  x5  x6 


©201 1  Air  Academy  Associates,  LLC.  Do  Not  Reproduce.  Page  81 


Academy 

Associates 


CPM  is  a  systems  engineering  best  practice  that  is  extremely  useful  in 
managing,  analyzing,  and  reporting  technical  product  performance.  It  is 
also  very  useful  in  decomposing  COIs  and  developing  linkages  between 
measures  and  task  capabilities/system  attributes. 


‘The  System  Can....'1 


-AAA 

h 

AAA 

m 


a  AAA 

A  S 


Vs 
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The  Optimize  Phase 
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The  DFSS  Process:  Identify,  Design, 

Optimize,  Validate 

-  The  Identify  Phase 

-The  DFSS  Scorecard 
-Voice  of  the  Customer  (VOC) 

-  The  Design  Phase 

-Translating  the  VOC  (Requirements  Flowdown) 

-Concept  Generation  and  Selection 

-Transfer  Functions 

-Critical  Parameter  Management 

f 

V 

-The  Optimize  Phase 

-Multiple  Response  Optimization 
-Expected  Value  Analysis  (Monte  Carlo  Simulation) 
-Parameter  (Robust)  Design 
-Tolerance  Allocation 

AIR* 

Academy 

Associates 

Simplify,  Perfect,  Innovate 

-  The  Validate  Phase 

-High  Throughput  Testing 
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Multiple  Response  Optimization 
Simulation*  Example 


Cross  Support 

Diameter  (2-4) 

Design  Cost  (<  300) 

Vertical  Support 

Diameter  (2-4) 

Vehicle 

Impact 

Rib  Compression  (<  11) 

Safety 

f 

Box  Diameter  (2-4) 

Process 

T 

Triaxial  Acceleration  (<  15) 

Beam  Angle  (35-60) 

AIR* 

Academy 

Associates 

-TluuWV^lA  I  ICO 

Simplify,  Perfect,  Innovate 

*  From  SimWare  Pro  by  Philip  Mayfield  and  Digital  Computations 
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Multiple  Response  Optimization  (cont.) 

Capability  Prior  to  Optimization 


Input  Controls 


Control  Set  1 , 


G.OOO 


3 
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Cross  Support  Diameter  (2  to 
41 


TOO  3  !200 

Vertical  Support  Diameter  [2  to 
A) 


vj  |40.00  ~^j 

Box  Diameter  (2  to  A)  Beam  Angle  (35  to  60) 


Simplify,  Perfect,  Innovate 


Multiple  Response  Optimization  (cont.) 

Capability  After  Optimization 


.poll  Cpk 


D  rib. compression  Cpk 


triaxial_accele  ration  Cpk  - 


±!  Cl 


270  DO  290-QQ  310  00  330  00 


<  > 


eao  0  50  iD  2o  ioso 


<  > 


Input  Controls 


Control  Set  1 


2.270 


Cross  Support  Uiameter  (2  to 
A) 


|3.78  3  14.00 

Vertical  Support  Uiameter  ( 'Z  to 
A) 


3  3 

Box  Diameter  (2  to  A)  Beam  Angle  (35  to  60) 
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DFSS  with  Monte  Carlo  Simulation 

•  Expected  Value  Analysis 

•  Robust  (Parameter)  Design 

•  Tolerance  Allocation 
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Expected  Value  Analysis  (EVA) 


EVA  is  the  technique  used  to  determine  the  characteristics  of  the  output 
distribution  (mean,  standard  deviation,  and  shape)  when  we  have 
knowledge  of  (1)  the  input  variable  distributions  and  (2)  the  transfer 
functions. 
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Simplify,  Perfect,  Innovate 


Expected  Value  Analysis  Example 


What  is  the  mean  or  expected  value  of  the  y  distribution? 
What  is  the  shape  of  the  y  distribution? 
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Parameter  Design  (Robust  Design) 

i 

L 

If  you"re  the 

T 

designer, 

1 

which  setting 

for  X  do  you 

prefer? 

l 

- h - *  x 

X, 

x2 

|  i 

k 

Changing  the  mean 

••••• 

- -X — i - 

^  °f  an  input  may 

— -y- -j^p-  possibly  reduce  the 

. f  i  1 

i  i  output  variation! 

- 1  /  i — - 

- — *  X 

X, 

x2 
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Robust  (Parameter)  Design 
Simulation*  Example 


Plug  Pressure  (20-50) 


Bellow  Pressure  (10-20) 


Ball  Valve  Pressure  (100-200) 


Water  Temp  (70-100) 


*  From  SimWare  Pro  by  Philip  Mayfield  and  Digital  Computations 
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Nuclear 

Reservoir 

Level 

Control 

Process 


Reservoir  Level  (700-900) 
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■input  Controls 


Control  Sell 


(500 
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472.727272727273  581.818181818182  690909090909091 


909.090909090909 


|S  Current  Data 
n  =  159 
Mean  =  681.9 
Standard  Deviation  =  141.6 
LSL  =  700 
USL  =  900 
Cp  =  0.2354 
Cpk  =  -0.0427 
DPM  =  612,650 
Sig  Cap  =  1.214 


SiU 


~j]  |12j00 


3  |145 


3 


Plug  Pressure  (20  to  50) 


Bellow  Pressure  (1 0  to  20) 


Ball  Valve  Pressure  (1 00  to  200) 
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70  '^r  100 

Water  Temp  (Expensive  to  Contrd) 


Simplify,  Perfect,  Innovate 
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-  Current  Data 
n  =  150 
Mean  =  802 

Standard  Deviation  =  21.58 
LSL  =  700 
USL  =  900 
Cp- 1.546 
Cpk  =  1.518 
DPM  =  3.829 
Sig  Cap  =  5.975 
0  Memorized  Data 
n  =  159 
Mean  =  681.9 
Standard  Deviation  =  141.6 
LSL  =  700 
USL  =  900 
Cp  =  0.2354 
Cpk  =  -0.0427 
DPM  =  612,650 
Sig  Cap  =  1.214 


Input  Controls 

Control  Set  1 


85 


3 


Plug  Pressure  (20  to  50) 


Bellow  Pressure  (1 0  to  20) 


Ball  Valve  Pressure  (1 00  to  200) 


Water  Temp  (Expensive  to  Control) 
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Tolerance  Allocation 


Which  input 
standard  deviations 
have  the  biggest 
effect  on  the  output 
variation? 
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Tolerance  Allocation  Example 


9  (R\  +  R2) 

H,  •  R2 


LSL  =  .255 
USL  =  .285 


Which  resistor's  standard  deviation  has  the  greater  impact  on  the 
capability  of  I? 
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Tolerance  Allocation  Example  (cont.) 


A  reduction  in  R/s  standard  deviation  (sigma)  significantly  reduces  the  dpm  while 
a  reduction  in  R2's  standard  deviation  has  a  smaller  effect. 


c. 


Tolerance  Allocation  Table 


. .  ...  --  • 

•  •  •  • '  1 

Factor 

Process  Inputs 

First 

Distro  Parameter 

Second 

Parameter 

R1 

Normal 

50 

2 

R2 

Normal 

100 

4 

A  reduction  in  R/s  standard  deviation  by  50%  (from  2  ohms  to  1  ohm)  combined 
with  an  increase  in  R2's  standard  deviation  by  25%  (from  4  ohms  to  5  ohms)  results 
in  a  dpm  =  9,743. 

(This  result  is  not  shown  in  the  table.) 
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Wall  thickness  (XI)  — 

CaO  mass  (X2) h 

H20  volume  (X3) h 


Y1=f1(X1,X2,  X3) 
Y2=f2(X1 ,  X2,  X3) 


-►  Time  to  use  (Y1) 


-►  Can  temp  (Y2) 


How  do  we  best  set  XI ,  X2,  X3  to  optimize  Y1  and  Y2? 

•  Expected  Value  Analysis  (EVA) 

-  a  form  of  Monte  Carlo  simulation 

•  Robust  Design  methods 

-  including  computer-based  Parameter  Design 

•  Tolerance  Allocation 

-  via  computer-based  tolerance  analysis 
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Process  Inputs 

1st 

Factor  Distro  Parameter 

2nd 

Parameter 

Exper 

Process  Output! 

Name  Function  LSL 

5 

USL 

Wall  Thickness  Normal  0  0.089  0 

Time  to  use  17.05547 

17 

CaO  Mass 

Normal 

□ 

0.0765 

0 

Max  can  ten 

106.9074 

107 

H20  Volume 

Normal 

□ 

0.04123 

0 

Noise Time  to  use 

Normal 

□ 

0.55075436 

0 

Noise Max  can  temp 

Normal 

□ 

0.29526055 

0 

*Y.: 


Expected  Value  Analysis 
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Time  (o  use  Histogram 


Process  Inputs 

First  Second 

Factor  Distro  Parameter  Parameter 


Wall  Thic  Normal 
CaO  Mas  Normal 
H20  Volu  Normal 
Noise_Tir  Normal 
Noise  Ms  Normal 


0.0S9 

0.0765 

0.04123 

0.550754 

0.295261 


Mat  can  temp  Histogram 


A 


Process  Outputs 

Time  to  use 

Max  can  temp 

#  of  Simulations 

1,000,000 

1,000.000 

Mean 

17.0426 

106.9269 

StdDev 

0.5554 

0.3073 

LSL 

USL 

17 

107 

Normal  Distro  Statistics 

KS  Test  p-Value  (Normal) 

0.211 

0.217 

dpm 

530,572.305 

405,965.585 

Cpk 

-0.026 

0.079 

Cp 

Sigma  Level 

-0.077 

0.23S 

Sigma  Capability 

-0.077 

0.238 
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Parameter  Ranges 


Wall  Thickness 
CaO  Mass 
H20  Volume 


Select  the  range  for  each  input  parameter 

From  |Q  to  pj 


Mean 

Mean 

Mean 


From  pT 


to  pr 


From  pT 


to  [pr 


N  oise_T  ime  to  use  M  ean 
Noise  Man  can  Mean 


From  pP 


to  pr 


From  p- 


to  pr 


Expected  Value  Analysis 


I  Optimize 
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Process  Inputs 

First  Second 
Distro  Parameter  Parameter 


Wall  Thic  Normal  -1  0.089 

CaO  Mas  Normal  -0.999978  0.0765 

H20Volu  Normal  -0.934036  0.04123 

Noise_Tir  Normal  0  0.550754 

Noise_Mc  Normal  0  0.295261 


Process  Outputs 


#  of  Simulations 

Mean 

StdDev 

LSL 

USL 


Normal  Distro  Statistics 


KS  Test  p- Value  (Normal) 

dpm 

Cpk 

Cp 

Sigma  Level 
Sigma  Capability 


Time  to  use 

1,000,000 

13.9331 

0.7062 

17 

0.0 

7.039 

1.448 

4.343 

4.343 


Computing  Optimal  Paramete 


*  #•'. 


Inputs 


Calculating 


Total  Weight  =  100 

Outputs 


Wall  Thickness  =  -1.0 
CaO  Mass  =  -0.99998 
H20  Volume  =  -0.93404 
Noise_Time  to  use  =  0.0 
Noise_Max  can  temp  =  0.0 


Time  to  use  =  4.445  dpm 
Max  can  temp  =  0.03  dpm 


Stop  Parameter  Design 

Help 

Cancel 

<<  Back 

Next  >> 

Finish 

Mas  can  temp  Histogram 


Max  can  temp 

1,000,000 

105.0354 

0.374 

107 

0.0 

.075 

1.751 

5.252 

5.252 
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Tolerance  Allocation  Table 


Process  Inputs 

First  Second 
Factor  Distro  Parameter 


Wall  Thick  Normal 
CaO  Mass  Normal 
H20  Volur  Normal 
Noise_Tim  Normal 
Noise  Mas  Normal 


-1 

-0.9999732 

-0.9340357 

0 

0 


0.039 

0.(14123 

0.55117544 

0.29*2605 


N  =  1,000,000 

Time  tr  use  1 

(in  dpmj 

rable  {Norn.^l  dpm) 

Wall  Thickness  CaO  Mass 

H20  Volume 

Noise  Time  to  use 

Noise  Max  can  temp 

r/d%  Sigma 

.1416 

\  6.722 

7.021 

.00198 

7.042 

-25%  Sigma 

1.023 

\7.009 

7.084 

.1988 

7.117 

|  -10%  Sigma 

3.26 

V.997 

7.096 

1.947 

7.117 

Nominal 

7.048 

r  .134 

7.056 

6.943 

6.989 

+10%  Sigma 

14.48 

/A  197 

7.074 

21.25 

7.055 

25%  Sigma 

40.3 

/  7.344 

7.065 

89.5 

6.995 

+5?%  Sigma 

171.0 

y  7.621 

7.085 

531.3 

7.061 

if.: 
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Expected  Value 


By  variance 
reduction 


Process  Inputs  ■  Process  Outputs 

First  Sect  nd 

Factor  Distro  Parameter  Parameter  Ht 


Time  (o  use  Histogram 


■l1'  ^  ^  ^  ^  ^  ^  '■ 


Wall  Thic  Normal 
CaO  Mas  Normal 
H20  Volu  Normal 
Noise_Tir  Normal 
NoisejVL  Normal 


#  of  Simulations 
an 

StdDe^ 

LSL 
USL 


Normal  Distro  Statistics 


KS  Test  p-Value  (Normal) 

cipm 

Cpk 

Cp 

Sigma  Level 
Sigma  Capability 


Time  to  use 

1,000,000 

13.9396 

0.6279 

17 

0.012 
.546 
1.625 

4.S74 
4.S74 


Max  can  temp 
1,000.000 
105.0247 
0.357S 

107 

0.0 
.017 
1 .84 

5.521 

5.521 
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The  DFSS  Process:  Identify,  Design, 

Optimize,  Validate 

-  The  Identify  Phase 

-The  DFSS  Scorecard 
-Voice  of  the  Customer  (VOC) 

-The  Design  Phase 

-Translating  the  VOC  (Requirements  Flowdown) 
-Concept  Generation  and  Selection 
-Transfer  Functions 
-Critical  Parameter  Management 

f 

V 

-  The  Optimize  Phase 

-Multiple  Response  Optimization 

-Expected  Value  Analysis  Using  Monte  Carlo  Simulation 

-Parameter  Design 

-Tolerance  Allocation 

AIR* 

Academy 

Associates 

Simplify,  Perfect,  Innovate 

-The  Validate  Phase 

-High  Throughput  Testing 
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The  Validate  Phase 


*  Validating  performance 

*  Performing  sensitivity  analysis 

*  Comparing  Predicted  capability  with  actual 

*  Gap  analysis  (reasons  for  lack  of 
confirmation) 

*  Updating  scorecards 
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Validate 

Critical  parameters 
are  validated 
against  predictions 
from  models. 

4, 

L$ 

Methods  mav 

•  Prototypes 

•  Lab  scale  p 

•  Test-fixturir 

If  validation  is  poor . 

BL  T  U 

include 

>roduction 

ig  of  subassemblies 

. gap  analysi 

SL 

is! 
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Introduction  to  High  Throughput  Testing 

(HTT) 

•  A  recently  developed  technique  based  on  combinatorics 

•  Used  to  test  myriad  combinations  of  many  factors  (typically  qualitative) 
where  the  factors  could  have  many  levels 

•  Uses  a  minimum  number  of  runs  or  combinations  to  do  this 

•  Software  (e.g.,  ProTest)  is  needed  to  select  the  minimal  subset  of  all  possible 

combinations  to  be  tested  so  that  all  2-way  combinations  are  tested. 

•  HTT  is  not  a  DOE  technique,  although  the  terminology  is  similar 

•  A  run  or  row  in  an  HTT  matrix  is,  like  DOE,  a  combination  of  different  factor 

f 

levels  which,  after  being  tested,  will  result  in  a  successful  or  failed  run 

•  HTT  has  its  origins  in  the  pharmaceutical  business  where  in  drug  discovery 
many  chemical  compounds  are  combined  together  (combinatorial  chemistry) 
at  many  different  strengths  to  try  to  produce  a  reaction. 

AIR* 

Academy 

Associates 

Simplify,  Perfect,  Innovate 

•  Other  industries  are  now  using  HTT,  e.g.,  software  testing,  materials 

discovery,  integration  and  functionality  testing  (see  example  on  next  page). 
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Submarine  Threat  Detection  Example 


■Suppose  we  want  to  perform  a  verification  test  with  the  following  7 
input  factors  (with  their  respective  settings): 


•Submarine Type  (SI,  S2,  S3) 

•Ocean  Depth  (Shallow,  Deep,  Very  Deep) 

•Sonar  Type  (Active,  Passive) 

•Target  Depth  (Surface,  Shallow,  Deep,  Very  Deep) 

•Sea  Bottom  (Rock,  Sand,  Mud) 

•Control  Mode  (Autonomous,  Manual) 

•Ocean  Current  (Strong,  Moderate,  Minimal) 

■All  possible  combinations  would  involve  how  many  runs  in  the  test? 


Academy 

Associates 


■If  we  were  interested  in  testing  all  pairs  only,  how  many  runs  would 
be  in  the  test?  Pro  Test  generated  the  following  test  matrix. 
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Submarine  Threat  Detection  Example  (cont.) 


The  following  15  test  cases  will  test  all  pairwise  combinations. 


Factor A 

Factor B 

Factor C 

Factor D 

Factor E 

Factor F 

Factor G 

Factor 

Name 

Submarine  Type 

Ocean  Depth 

Sonar  Type 

Target  Depth 

Sea  Bottom 

Control  Mode 

Ocean  Current 

Case  1 

S3 

Deep 

Passive 

Very  Deep 

Mud 

Manual 

Minimal 

Case  2 

SI 

Very  Deep 

Passive 

Surface 

Rock 

Autonomous 

Strong 

Case  3 

S2 

Shallow 

Active 

Shallow 

Rock 

Manual 

Moderate 

Case  4 

S2 

Deep 

Passive 

Deep 

Sand 

Autonomous 

Moderate 

Case  5 

SI 

Shallow 

Active 

Surface 

Sand 

Manual 

Minimal 

Case  6 

SI 

Very  Deep 

Passive 

Shallow 

Mud 

Autonomous 

Minimal 

Case  7 

S3 

Very  Deep 

Active 

Deep 

Mud 

Manual 

Strong 

Case  8 

S2 

Very  Deep 

Active 

Very  Deep 

Sand 

Autonomous 

Strong 

Case  9 

S3 

Shallow 

Passive 

Shallow 

Mud 

Autonomous 

Strong 

Case  1 0 

S3 

Deep 

Active 

Surface 

Rock 

Manual 

Moderate 

Case  11 

SI 

Shallow 

Active 

Deep 

Rock 

Autonomous 

Minimal 

Case  1 2 

SI 

Deep 

Passive 

Very  Deep 

Rock 

Manual 

Moderate 

Case  13 

S2 

Very  Deep 

Active 

Surface 

Mud 

Autonomous 

Moderate 

Case  1 4 

S3 

Deep 

Active 

Shallow 

Sand 

Manual 

Strong 

Case  15 

S2 

Shallow 

Active 

Very  Deep 

Rock 

Manual 

Minimal 
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HTT  Applications 


•  Reducing  the  cost  and  time  of  testing  while  maintaining 
adequate  test  coverage 

•  Integration  and  functionality  testing 

•  Creating  a  test  plan  to  stress  a  product  and  discover  problems 

•  Prescreening  before  a  large  DOE  to  ensure  all  2-way 
combinations  are  feasible  before  discovering,  midway  through 
an  experiment,  that  certain  combinations  are  not  feasible 

•  Developing  an  — oter  array”  of  noise  combinations  to  use  in  a 
robust  design  DOE  when  the  number  of  noise  factors  and 
settings  is  large 
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Requirements  Flowdown  Using  QFD 


Functional  Req. 
(CTC's! 


Academy 
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Customer 

Expectations 


cr 

0 

House  of 

0 

E 

o 

Quality 

0 

3 

#1  ' 

O 

T 

Prod.  Design  | 
Characteristics  1 

Functional  Req. 

(CTC's) 

House  of 
Quality 
#2 

* 


Product 
Design  CTC's 


Mfg.  Process 
Characteristics 

House  of 
Quality 
#3 


Marketing 

•  Features 

•  Quality 

•  Performance 

•  Cost 


Design  Engineering 

•  Performance 

•  Reliability 

•  Cost 


Mfg.  Engineering 

•  Manufacturability 

•  Cost 


Mfg.  Process 
Control 

Mfg.  Process 

Characteristics 

House  of 
Quality 
#4 

J 

n 

L 

Mfg.  Process 
Control 


Manufacturing 

•  SPC 

•  Process  Capability 


©2011  Air  Academy  Associates,  LLC.  Do  Not  Reproduce. 


Page  1 1 2 


Simplify,  Perfect,  Innovate 


Validate 
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Critical  parameters 
are  validated 
against  predictions 
from  models. 


LSL 


USL 


Methods  may  include 

•  Prototypes 

•  Lab  scale  production 

•  Test-fixturing  of  sub-assemblies 
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Validate 
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PPs 

(Process  Domain) 


DP 

(Physical  Domain) 


Wall  thickness 

3 

9 

3 

3 

CaO  mass 

5 

3 

9 

H20  volume 

4 

Design  Parameters  are 
mapped  to  Process 

Button  force 

2 

■ 

Parameters 

■ 

Prioritized  PPs 

75 

69 

50 

62 

17 

- 
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Clamp  Force 
Injection  Pressure 
Seamer  roll  force 


■ 

Prioritized  PCs 

67 

71 

19 

45 

67 

. 

©2011  Air  Academy  Associates,  LLC.  Do  Not  Reproduce. 


Page  1 1 5 


Simplify,  Perfect,  Innovate 


Identify 


Design 


Optimize 


Validate 


Mfg. 


QFD 


Axiomatic 

Design 


TRIZ  Analytical  Modeling  LSS/DFSS 
&  Simulation 


voc 

i 

CUSTOMER  DOMAIN 

8  PATTERNS 

SURVEYS 

INTERVIEWS 

HOQ1 

t 

FUNCTIONAL  DOMAIN 

SYSTEMS  VIEW 

FOCUS  GROUPS 

CTCs  (FPs) 

PAIRWISE  COMPARISON 
BASES 

CTCs  (FPs) 
_ ±_ _ 

FUNCTIONAL  DOMAIN 

i 

FUNCTIONAL  MODEL 

Functional  Analysis 

HOQ2 
- ^ - 

(VIA  AXIOMATIC  DESIGN) 

TC&  PC  ALGORITHMS 

System  Technique 

DPs 

PHYSICAL  DOMAIN 

RESOURCES 

FEA,  CFD: 
ANSYS 

(FAST) 

INDEPENDENCE 

FUNCTIONAL  MODEL 

FLUENT 

DOE  /  EVA 

DPs 

&  INFORMATION 
OPTIMIZATION 
(DECOUPLING) 

TC&  PC  ALGORITHMS 

COSMOS 

MONTE  CARLO  /  DS 
PARAMETER  DESIGN 
TOLERANCING 

DPs 

± 


HOQ3 

PPs 


PPs 

i 


HOQ4 

PC 


PHYSICAL  DOMAIN 

t 

PROCESS  DOMAIN 

TC&  PC  ALGORITHMS 

CONFIRMATION 

PROCESS  DOMAIN 

t 

PROCESS  CONTROL 

TC&  PC  ALGORITHMS 

HYPOTHESIS  TESTS 
CONFIRMATION 


SPC 

CONTROL  PLANS 
POKAYOKE 
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The  Original  DFSS  (Design  for  Six  Sigma) 


Identify 


FUNCTIONAL 

REQUIREMENTS 

CUSTOMER 

REQUIREMENTS 

HOQ1 

Design 


DESIGN 

PARAMETERS 

FUNCTIONAL 

REQUIREMENTS 

HOQ2 

Optimize 
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MANUFACTURING 

PARAMETERS 

DESIGN 

PARAMETERS 

HOQ3 

9 

Validate 
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o 

Z  (0 

5  a: 

o«u 
<  i 
£2 
12 


PROCESS  CONTROL 
PARAMETERS 


HOQ4 


CO 

h 

oc  z 

LU  LU 

s  § 
o  w 

H  O' 
co  =: 


LU 

or 


FUNCTIONAL 

REQUIREMENTS 


HOQ1 


DFR 


RELIABILITY 

TESTS 


HOQ5 
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DESIGN 

PARAMETERS 

FUNCTIONAL 

REQUIREMENTS 

HOQ2 

DFSS, 


In  DFSS0we  use  H0Q1-4 
In  DFR  we  add  H0Q5  &  H0Q6 

•  In  HOQ5  we  list  and  prioritize 
Reliability  Tests  which  address 
each  Functional  Requirement. 

•  In  HOQ6  we  list  and  prioritize 
Test  Procedures  which  address 
each  Reliability  Test. 


CO 

O' 

z  w 

g  t 

CO  § 

W  < 
o 


MANUFACTURING 

PARAMETERS 


HOQ3 


Product 

Validation 


Functional 

Validation 


TEST 

PROCEDURES 

RELIABILITY 

TESTS 

HOQ6 

PROCESS  CONTROL 
PARAMETERS 

MANUFACTURING 

PARAMETERS 

HOQ4 

©2011  Air  Academy  Associates,  LLC.  Do  Not  Reproduce. 


Page  1 1 8 


MBT&E  with  Design  for  Successful  Systems 


PERFORMANCE 

MEASURES 


Use  these  matrices  of  DFSS 
for  Designing  the  T&E 
(see  next  page) 
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Steps  for  Designing  the  Test  and  Evalution* 

Tools  and  Methods  from  DFSS  that  can  help  accomplish  these 

steps  are  in  parentheses: 

•  Develop  the  measures  of  effectiveness  from  the  task  capabilities  and 
the  measures  of  performance  from  the  system  attributes.  (HOQ  1) 

•  Determine  the  operational  factors  and  conditions.  (HOQ  2) 

•  Develop  linkages  between  measures  and  COIs.  (CPM) 

•  Complete  linkages  from  measure-to-system-to-task.  (CPM) 

•  Assign  one  or  more  data  sources  to  each  evaluation  measure.  (HOQ  5) 

•  Determine  the  operational  conditions  that  can  or  cannot  be  addressed 
by  the  identified  data  sources.  (HOQ  2,  CPM,  and  HOQ  5) 

•  Develop  detailed  measure  design.  (HOQ  6) 

•  Develop  design  of  experiments.  (HOQ  2,  CPM,  HOQ  5,  HOQ  6) 

•  These  steps  are  taken  from  Chris  Wilcox's  MBT&E  Tutorial  (page  25)  at  NDIA  T&E  2010. 
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Partial  Listing  of  Who  Has  Used  Our 
DFSS  Process  and  Tools 


•  Xerox 

•  Gates  Rubber  Company 

•  Hyundai 

•  Timken 

•  GE  Medical  Systems 

•  Medtronic 

•  St.  Jude  Medical 

•  Sony 

•  John  Deere 

•  Delphi 

•  Sensis 


Nokia 

Bose  Corporation 
•PerkinElmer 
Samsung 
ATM  I 

Poliak  Industries 

Sandia  National  Laboratory 

Abbott  Laboratory  Diagnostics 

GlaxoSmithKline 

General  Dynamics  Land 
Systems 
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GE's  First  DFSS  System  C98): 

Full  Use  of  Six  Siqma/DFSS  Tools 

Key  customer  CTQs  identified  — 

-  Image  quality 

-  Speed 

-  Software  reliability 

-  Patient  comfort 

Disciplined  systems  approach:  90  system  CTQs 
33  Six  Sigma  (DMAIC)  or  DFSS  projects/studies 
Scorecard-driven 

Part  CTQs  verified  before  systems  integration 


Leading-Edge  Technology 

World's  first  1 6-row  CT  detector 
Multi-slice  data  acquisition 
64-bit  RISC  computer  architecture 
Long-life  Performix™  tube 


Results 

Better  image  quality 

-  Earlier,  more  reliable  diagnoses 

-  New  applications:  vascular  imaging,  pulmonary  embolism, 
multi-phase  liver  studies,... 

•  Much  faster  scanning: 

-  Head:  from  1  min  to  19  sec  (9  million/yr) 

-  Chest/abdomen:  from  3  min  to  17  sec  (4  million/yr) 

Clinical  productivity  up  50% 
lOx  improvement  in  software  reliability 
Patient  comfort  improved  -  shorter  exam  time 
Development  time  shortened  by  2  years 
High  market  share;  significant  margin  increase 


^'Biggest  breakthrough  in  CTjn  a  decade,^  Gary  Glazer,  Stanford 


©2011  Air  Academy  Associates,  LLC.  Do  Not  Reproduce. 


Page  1 24 


Academy 

Associates 


Simplify,  Perfect,  Innovate 


Sigma/DFSS  Financial  Benefits: 

96-  00 
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Margin  on  revenue  growth  from 
customer  delight,  including 
DFSS  products,  ~  $1000M  3000 
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Major  impact  on  the  bottom  line 

Significant  benefits  from  customer  delight,  including  DFSS 
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Xerox  Develops  New  Paper 

Wall  Street  Journal: 

Xerox  Develops  a  „Green"  Paper,  But  Will  Firms  Add  it  to  Fold? 

By  William  M.  Bulkeley 

July  30,  2007;  Page  B3 

Xerox  has  invented  an  environmentally  friendly  copy  paper  that  costs  less.  The 
new  cut-sheet  “High-Yield  Business  Paper”  requires  half  as  many  trees,  fewer 

f 

chemicals  and  less  energy  to  manufacture  and  it  weights  less,  reducing  postage 
and  trucking  costs.  Merilyn  Dunn  of  InofTrends  suggests  the  paper  will  be  used 
for  transactions  such  as  invoices  and  phone  bills  where  people  don't  care  about 
long-term  archiving  of  documents.  Xerox  and  others  have  tried  to  use  cheap 

newsprint  in  copiers  and  laser  printers  in  the  past,  but  “you  always  had 
catastrophically  bad  results  related  to  the  curl  in  a  digital  printer,”  said  Steve 

AIR^ 

Simpson,  Xerox"s  vice  president  in  charge  of  paper  and  supplies.  Bruce  Katz,  a 
paper  technologist  in  Xerox"s  research  facility  in  Webster,  said  he  was  able  to 
overcome  the  curling  problem  by  figuring  out  how  to  make  cellulose  fibers  in  the 
paper  line  up  evenly,  so  they  would  shrink  at  the  same  rate  when  the  toner  fusing 

process  took  place. 

Academy 

Associates 

Note:  Bruce  Katz,  a  Xerox  DFLSS  GB,  used  the  DesIgNNOVATION™  methods  to  accomplish  this. 

Simplify,  Perfect,  Innovate 
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Photoreceptor  Belt  Tensioning  System 


iSixSigma  Magazine 
July/August  2007,  pp  47-55 
By  Bob  Hildebrand,  Xerox  DFLSS  Black  Belt 


The  Xerox  Corp.  designs,  manufactures  and  markets  iGen3,  a  color 
printer  that  can  produce  photo-quality  prints  at  1 10  pages  per  minute. 
When  the  current  iGen3  was  to  be  modified,  the  engineering  team  was 
tasked  with  redesigning  the  belt  tensioning  mechanism  on  the 
photoreceptor  into  a  smaller  package  without  adjusting  the  length  of  the 
belt.  The  redesign  had  to  take  several  noise  factors  into  account.  The 
outcome  of  the  project  was  a  design  that  met  the  constraints  placed  on  it 
by  the  system.  This  IDOV  project  is  a  practical  example  of  how  Design 
for  Lean  Six  Sigma  (DFLSS)  can  bring  about  the  best  option  available  in 

a  constrained  design. 

Please  see  the  referenced  article  for  a  detailed  presentation  of  this  case  study. 


©201 1  Air  Academy  Associates,  LLC.  Do  Not  Reproduce.  Page  1 27 


Academy 

Associates 


Simplify,  Perfect,  Innovate 


Some  Results  From  Other  DFSS  Studies 


Accelerated  Testing  of  a  Proprietary  Product 

-  Time  to  qualify  process  changes  reduced  from  a  year  to  5  weeks  -  860%  test  cost  reduction 
-  5  years  benefit  of  $48.5M  based  on  accelerated  placement  of  lower  cost  units 

Regression  Analysis  to  Predict  Life  of  a  Proprietary  Product 

-  $2M  ANPV  Improvement 
-  24  hours  to  develop  right  material 

-  Overall  length  of  project:  3  months  (vs.  2  years  using  traditional  approach) 

-  Life  expectancy  improvement:  over  4x! 

Modeling  to  Reduce  Development  Costs  and  Improve  TTM 

-  Matured  the  new  design  to  last  for  >5  Million  cycles  in  6  months 

-  Demonstrated  that  following  DFSS  can  accelerate  Time  to  Market 

-  Established  the  importance  that  all  QMS  parts  go  through  the  DFSS  process 

Identifying  Critical  Parameters 

-  25%  cost  reduction  of  part:  $3M  savings 

-  Leveraged  the  new  accurate  measuring  process  across  product  lines 

-  Short  term  solution  in  two  months,  long  term  took  a  year 

Supply  Problem  Resolution  Using  Simple  Hypothesis  Testing 

-  $2M  immediate  savings  and  saved  the  product  from  being  withdrawn  from  field 
-  Took  just  four  months  to  resolve  a  problem  that  had  lingered  for  10  years 

-  Gained  control  of  infant  mortality  (i.e.,  failures  within  first  6  months) 
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Using  DFSS  to  Improve  Reliability  Growth 


FEF  =  Fix  Effectiveness  Factor 

Historical  data  from  reliability  growth  models  indicates  an  overall 
average  of  .7 

(Source:  Larry  Crow‘s  RAMS  2011  presentation,  page  68) 

Using  a  DFSS  FEF  of  at  least  .9,  we  can  see  that  the  number  of 
iterations  can  be  reduced  substantially  to  achieve  the  same  goal. 
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For  More  Information,  Please  Contact 


Mark  Kiemele 

Air  Academy  Associates,  LLC 

1650  Telstar  Drive,  Ste  110 
Colorado  Springs,  CO  80920 


Toll  Free:  (800)  748-1277  or  (719)  531-0777 
Facsimile:  (719)  531-0778 
Email:  aaa@airacad.com 
or  mkiemele@airacad.com 
Website:  www.airacad.com 
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Advanced  Range  Data  System  (ARDS) 
Service  Life  Extension  Program  (SLEP) 

“ Ensuring  GPS  Based  TSP I  Remains  a  Viable  T&E  Range 

Instrumentation  Asset  ” 

By 

V - 

Mr.  Dick  Dickson 

Tri-Service  GPS  Sustainment  Management  Office  IPT  Lead 

TYBRIN  Corporation 

Presented  at 
NDIA 

27th  ANNUAL  NATIONAL  TEST  &  EVALUATION  CONFERENCE 


March  2011 
Tampa,  FL 


ARDS  SLEP 
Background 


T/BRIN 


•  The  Advanced  Range  Data  System  (ARDS)  is  a  GPS  based  TSPI 
instrumentation  suite  originally  fielded  in  the  early  1990’s. 

-  Full  Scale  Engineering  Development  (FSED)  and  Low  Rate  Initial 
Production  (LRIP). 

•  Full  Rate  Production  (FRP)  hardware  was  fielded  in  1997-1998. 

-  Total  investment  including  all  CTEIP  and  l&M  funding  from  conception  to 
FRP  hardware  was  just  over  $500M. 

•  The  expected  life  span  was  8-10  years. 

-  Production  hardware  was  delivered  with  numerous  components  already 
deemed  obsolete  requiring  immediate  obsolete  component  replacement 
programs. 

•  The  initial  effort  to  retrofit  and  upgrade  the  system  in  1998-1999 
alleviated  the  obsolete  component  issues  present  when  the  FRP 
hardware  was  delivered. 
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ARDS  SLEP 
Background 


T/BRIN 


•  Preparations  for  a  new  follow  on  CTEIP  program  called  the 
Enhanced  Range  Applications  Program  (EnRAP)  began  in  2001. 

•  This  program  was  targeted  at  providing  significant 
enhancements  and  improvements  to  the  existing  ARDS 
hardware  suite 

-  Improved  performance  and  TSPI  solution  accuracy. 

-  Significant  component  miniaturization. 

-  More  efficient  data  link  system. 

•  EnRAP  hardware  was  supposed  to  be  the  next  generation  GPS 
TSPI  hardware  suite  that  would  replace  ARDS  starting  in  2007. 

-  Timed  to  be  fielded  at  the  end  of  the  original  10  year  expected  service  life 
of  the  ARDS  full  rate  production  hardware  suite. 

-  The  contract  was  awarded  in  2005. 


ARDS  SLEP 
Background 


T/BRIN 


•  The  EnRAP  program  started  experiencing  problems  shortly  after 
it  began  and  was  canceled  in  March  2006. 

•  The  T&E  ranges  involved  in  the  EnRAP  program  immediately 
initiated  a  Service  Life  Extension  Program  (SLEP)  on  the  ARDS 
hardware  suite. 

•  T&E  ranges  involved  in  the  ARDS  SLEP  established  the 
following  objectives  and  goals. 

-  Replace  the  obsolete  and  soon  to  be  obsolete  components  identified  in 
the  most  cost  efficient  manner  possible. 

-  Develop  form-fit-function  replacements  where  possible. 

-  If  not  form-fit-function,  develop  replacements  that  required  the  least 
amount  of  changes  to  the  system  overall  (wiring  harnesses,  mounting  rail, 
software  mods,  etc.). 

-  Procure  life  time  supply  of  parts  identified  as  soon  to  be  obsolete. 
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WBRIN  ARDS  SLEP  GOALS 


•  Focus  on  replacing  obsolete  hardware  with  equivalent 
capabilities. 

-  Enhancements  and  improvement  in  performance  not  the  primary 
goal. 

-  Maintain  current  performance  capabilities  as  a  minimum. 

-  Driven  by  what  the  available  funding  was  allowed  to  be  spent  on. 

•  Eliminate  proprietary  hardware  and  software  wherever 

possible. 

•  Develop  multiple  sources  of  procurement  for  key 

components. 


ARDS  SLEP 
Background 


T/BRIN 


•  The  ARDS  SLEP  began  officially  in  FY07. 

-  The  majority  of  the  SLEP  efforts  are  being  executed  through  the  Tri- 
Service  GPS  Sustainment  Management  Office  (GPS  SMO)  out  of  the 
Naval  Air  Warfare  Center  Weapons  Division,  China  Lake. 

-  There  are  two  key  in-house  obsolete  component  replacement  efforts 
being  executed  by  the  46th  TW  at  Eglin  AFB. 

•  Replacement  for  the  Advanced  Digital  Interface  Unit  (ADIU) 

•  Replacement  for  the  Intelligent  Flash  Solid  Sate  Recorder  (IFSSR) 

-  Key  T&E  ranges  involved  are  China  Lake,  Pax  River,  Eglin  AFB,  Edwards 
AFB,  and  White  Sands  Missile  Range  (WSMR). 

-  Also  involved  are  two  German  T&E  ranges  that  have  the  same  hardware 
fielded  and  operational. 
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ARDS  SLEP 
Background 
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•  The  ARDS  SLEP  is  divided  up  into  several  areas. 

-  Development  of  new  hardware  to  replace  obsolete  hardware  where  no 
current  off-the-shelf  solution  exists. 

-  Life  time  buy  of  hardware  that  will  soon  be  obsolete. 

•  Develop  multiple  sources  of  procurement  for  key  components. 

-  DLT  -  Modem  and  Power  Amplifier 

-  ADIU 

-  IFSSR 

-  Power  Supplies  -  Red  and  Black 

-  ARDS  Pod  Cable  Harnesses 

-  GPS  Receiver 
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ARDS  SLEP 
Components  Involved 
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•  ARDS  components  being  addressed  in  the  current 
SLEP. 

-  Data  Link  Transceiver  (including  the  modem  and  power  amplifier). 

*  Procure  new  backwards  compatible  modems  from  DRS  Defense 
Solutions. 

*  Procure  new  replacement  DLT  power  amplifiers  from  DRS  Defense 
Solutions  (developed  by  Aethercomm  to  a  DRS  specification). 

*  Develop  and  procure  a  replacement  DLT  power  amplifier  to  the 
current  government  owned  SCD  from  Nanowave  Technologies  (the 
original  manufacturer). 

*  Take  the  government  owned  Multi-Service  Target  Control  System 
(MSTCS)  DLT  and  migrate  it  to  a  fully  ARDS  compatible  DLT  (form- 
fit-function). 

*  Develop  a  new  ARDS  compatible  miniaturized  DLT  for  use  in 
onboard  installations  in  the  JSF,  F-22,  UAV  applications,  etc. 
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ARDS  SLEP 
Components  Involved 


T/BRIN 


•  ARDS  components  being  addressed  in  the  current 
SLEP  (Cont...). 

-  AC/DC  Converter  (DLT  power  supply). 

*  Procure  more  of  the  current  power  supplies  from  Technipower  LLC 
(One  of  two  original  equipment  manufactures). 

-  GPS  receiver 

*  Procure  the  new  form-fit-function  replacement  DRS  Integrated  GPS 
System  (DIGS)  receiver  from  DRS  Defense  Solutions  to  replace  the 
obsolete  Rockwell  Collins  GNP-10. 

*  Develop  two  separate  NovAtel  commercial  receiver  solutions  to 
replace  the  GNP-10. 

-  Advanced  Digital  Interface  Unit  (ADIU). 

*  Procure  a  new  ADIU  from  DRS  Defense  Solutions. 

*  Develop  and  manufacture  a  new  government  owned  ADIU 


ARDS  SLEP 
Components  Involved 


T/BRIN 


•  ARDS  components  being  addressed  in  the  current 
SLEP  (Cont...) 

-  Intelligent  Flash  Solid  State  Recorder  (IFSSR) 

•  Procure  a  new  IFSSR  from  DRS  Defense  Solutions. 

•  Develop  and  manufacture  a  new  government  owned  IFSSR. 

-  DC/DC  Power  Supply 

•  Develop  a  new  DC/DC  power  supply. 

•  Competed  the  development  -  awarded  to  Technipower  (now  Unipower)  LLC. 

•  Government  developed  a  new  updated  equipment  specification 

-  New  Red,  Black,  and  REM  by-pass  Cable  Harnesses 

•  Develop  a  second  source  of  procurement. 

-  ARDS  Pod  Tube  Hangers  -  Forward,  Center,  and  Aft 
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ARDS  SLEP 
Current  Progress 


T/BRIN 


•  Replacement  DLT  power  amplifier  manufactured  by 
Nanowave. 

-  Nanowave  was  the  original  manufacture  of  the  Full  Rate  Production 
ARDS  DLT  power  amplifier. 

-  German  T&E  ranges,  via  the  Tri-Service  GPS  SMO  and  FMS  cases, 
funded  the  development  of  a  replacement  DLT  power  amplifier  built 
to  the  same  SCD  as  the  original  DLT  power  amplifier. 

•  The  initial  requirement  was  for  the  procurement  of  52 
new  DLT  power  amplifiers. 

-  Additional  orders  were  placed  bringing  the  total  ordered  to  105. 

-  All  105  power  amplifiers  have  been  delivered  and  accepted. 

•  The  DD  Form  1494  frequency  approval  process  has 
been  completed. 


ARDS  SLEP 
Current  Progress 


T/BRIN 


*  Replacement  DLT  power  amplifier  manufactured  by 
Aethercomm  and  sold  through  DRS  Defense  Solutions. 

-  Developed  by  Aethercomm  for  DRS  and  designed  to  a  new  DRS 
proprietary  specification. 

-The  DD  Form  1494  approval  process  has  been  completed. 

•  Multi-Service  Target  Control  System  (MSTCS)  DLT 
conversion  effort. 

-  The  MSTCS  DLT  was  developed  under  a  separate  CTEIP  program 
and  was  based  on  the  ARDS  DLT  architecture. 

-  The  government  owns  the  rights  to  the  MSTCS  DLT  hardware  and 
software  design  (modem  and  power  amplifier) 

-  Current  ARDS  SLEP  activities  include  funding  the  conversion  of  the 
MSTCS  DLT  to  operate  as  a  fully  compatible  ARDS  DLT. 
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ARDS  SLEP 
Current  Progress 


T/BRIN 


•  MSTCS  DLT  Conversion  Effort  (cont...) 

-  Conversion  efforts  include  repackaging  the  converted 
MSTCS  DLT  into  the  ARDS  form  factor. 

-  Delivery  includes  just  the  single  modem  CCA  and  a 
repackaged  power  amplifier. 

-  Government  engineers  at  the  46th  TW  at  Eglin  will  perform 
the  final  assembly  into  ARDS  modem  and  power  amplifier 
housings  -  form  factor. 

-  Government  is  performing  all  the  environmental  stress 
screening,  EMI/EMC,  and  shock/vibration  testing 
in-house. 
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ARDS  SLEP 
Current  Progress 


T/BRIN 


•  Miniaturized  ARDS  DLT  development  effort. 

-  The  T&E  ranges  developed  a  miniaturized  ARDS  compatible  DLT 
capability  to  instrument  smaller  test  platforms  several  years  back. 

-  The  “ARDS  Lite”  package  utilized  COTS  FreeWave  radios  that 
were  coupled  to  a  special  data  link  relay  (FMIU)  in  order  to  work  in 
the  ARDS  network. 

*  Obsolescence  issues  were  present  with  this  system  as  well. 

*  Allowed  only  1/3  of  the  RF  throughput  of  a  true  ARDS  DLT. 

*  Utilized  a  truncated  or  compressed  ARDS  message  format. 

*  Required  a  dynamic  translation  of  ARDS  messages  into  their 
ARDS  "Lite"  equivalents  (and  vice  versa)  in  real  time. 
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ARDS  SLEP 
Current  Progress 


T/BRIN 


•  Miniaturized  ARDS  DLT  development  effort  (Cont...). 

-  The  current  miniaturized  ARDS  DLT  development  effort  took  the 
existing  ARDS  “Lite”  transceiver  and  replace  it  with  a  ARDS 
capable  miniature  DLT  with  most  of  the  functionality  of  the 
current  DLT 

•  Does  not  currently  have  a  relay  capability. 

•  Utilizes  FI  frequency  only. 

•  Encryption  capability  not  currently  present. 

-  The  new  miniaturized  DLT  development  resulted  in  a  small,  low 
cost  ARDS  DLT. 

•  The  baseline  production  hardware  has  been  received  and  accepted  by 
the  government. 

•  The  DD  Form  1494  frequency  approval  process  is  in  work. 

-  Funded  improvements  to  the  baseline  product  include  adding  a  link-less 
capability  and  a  live  monitor  mode. 
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ARDS  SLEP 
Current  Progress 


T/BRIN 


•  AC/DC  Power  Supply. 

-  Technipower  LLC,  formerly  Transchem,  was  one  of  two  original 
manufactures  of  the  96150400  AC/DC  power  supply. 

•  The  second  manufacture  Keltec,  no  longer  manufactures  this  power 
supply. 

-  CM  approved  AC/DC  power  supply  for  ARDS. 

-  Source  Control  Drawing  in  the  ARDS  documentation 
package. 

-  Originally  thought  to  be  obsolete  and  out  of  production. 

-  This  power  supply  is  still  a  standard  production  line  for 
Technipower  in  accordance  with  the  government  owned  SCD. 

-  All  power  supplies  ordered  have  been  delivered  and  accepted. 
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ARDS  SLEP 
Current  Progress 


T/BRIN 


•  GPS  Receiver  Replacement. 

-  Current  GNP-10  GPS  receiver  manufactured  by  Rockwell  Collins 
is  obsolete  and  can  no  longer  be  procured. 

-  Failed  GNP-10  units  sent  in  for  repair  are  starting  to  be  returned 
“Beyond  Economical  Repair”  (BER). 

-  No  drop  in  replacement  available  from  Rockwell  Collins  without 
extensive  NRE  ($5-6M). 

-  DRS  Defense  Solutions  developed  a  GNP-10  replacement  for  use 
on  the  P-5  program. 

•  DRS  Integrated  GPS  System  (DIGS). 

•  NRE  was  covered  100%  by  other  DRS  programs  and  internal  R&D 
funding. 

-  The  T&E  ranges  were  able  to  procure  this  new  “form-fit-function” 
replacement  GPS  receiver  without  any  NRE  expenses. 
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ARDS  SLEP 
Current  Progress 


T/BRIN 


•  GPS  Receiver  Replacement  (Cont...). 

-  TYBRIN  initiated  contract  actions  to  procure  DRS  Defense  Solutions 
new  replacement  DIGS  GPS  receiver. 

•  Accuracy  and  performance  problems  were  discovered  during  several 
rounds  of  low  dynamic  truck  tests  and  flight  tests  conducted  at  Eglin. 

•  Problems  were  also  discovered  during  attempts  to  post  process  raw  data 
from  the  DIGS. 

-  DRS  has  made  significant  progress  in  resolving  the  problems 
identified. 

-  Low  dynamic  flight  testing  on  the  redesigned  DIGS  receiver  (new 
Kalman  filter)  has  been  completed. 

-  High  dynamic  accuracy  testing  is  tentatively  scheduled  for  April  201 1 . 
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ARDS  SLEP 
Current  Progress 


T/BRIN 


•  GPS  Receiver  Replacement  (Cont...). 

-  Navy  integration  of  the  Novatel  Synchronous  Position,  Attitude  and 
Navigation  (SPAN)  SE  GPS/INS  receiver. 

•  Worked  with  NovAtel  to  develop  and  evaluate  a  new  NovAtel  commercial 
receiver  GPS/INS  (LN-200)  instrumentation  package  to  replace  the 
current  GNP-10/LN-200  instrumentation  package. 

•  Designed  for  use  in  the  new  F/A-18  internal  mount  configuration. 

•  Not  designed  for  use  in  the  ARDS  pod. 

•  Objective  -  achieve  the  same  performance  and  TSPI  accuracy  as  the 
current  GNP-10/LN-200  configuration. 

•  The  new  GPS/INS  system  has  been  tested  and  evaluated  in  ground 
tests  (van),  low  dynamic  flight  testing  (Baron  prop  plane),  and  high 
dynamic  flight  testing  on  the  F/A-18. 

•  Successful  flight  testing  and  TSPI  accuracy  testing  has  been  completed. 

•  Development  of  production  internal  mount  configurations  is  underway. 
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ARDS  SLEP 
Current  Progress 


T/BRIN 


•  GPS  Receiver  Replacement  (Cont...). 

-  Air  Force  (Eglin)  46th  TW  Development  and  Integration  of  the  Eglin 
NovAtel  SPAN  GPS  Receiver  (ENGR). 

•  Worked  with  NovAtel  to  develop  and  evaluate  a  new  Kinematic  Carrier 
Phase  capable  NovAtel  SPAN  commercial  GPS  receiver  coupled  with  the 
LN-200  Inertial  Measurement  Unit  (IMU)  instrumentation  package  to 
replace  the  current  GNP-10/LN-200  TSPI  instrumentation  package. 

•  Repackaged  into  a  GNP-10  form  factor  -  ARDS  Pod  configuration. 

•  Objective  -  achieve  the  same  Method  I  performance  and  TSPI  accuracy 
as  the  current  GNP-10/LN-200  configuration. 

•  Outputs  both  GNP-10  format  messages  and  NovAtel  messages  via  USB 
or  Ethernet  -  NovAtel  messages  used  for  the  post  processing. 

•  Primary  difference  between  the  GNP-10  and  ENGR  is  that  the  ENGR  will 
accomplish  Differential  GPS  (DGPS)  via  WASS  corrections  versus  the 
RAJPO  DGPS  format. 

•  Dynamic  flight  testing  comparing  the  ENGR  performance  against  the 
GNP-10  and  other  TSPI  sources  is  underway  now. 
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ARDS  SLEP 
Current  Progress 


T/BRIN 


•  ADIU  and  IFSSR  development  efforts. 

-  Two  development  efforts  are  underway  for  a  replacement  ADIU  and 
IFSSR. 

-  The  46th  TW  at  Eglin  AFB  is  developing  a  replacement  ADIU  and 
IFSSR  in-house. 

•  All  hardware  and  software  design  will  be  owned  by  the  government. 

-  A  major  T&E  range  customer  procured  a  follow-on  DRS  developed 
replacement  for  the  ADIU  and  IFSSR  as  well. 

•  Schedule  requirements  dictated  that  the  replacements  would  be  needed 
before  the  in-house  government  effort  at  Eglin  would  be  completed. 

•  Hardware  and  software  for  the  DRS  development  will  be  proprietary  to 
DRS. 

•  All  hardware  ordered  has  been  delivered. 
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T/BRIN  ARDS  SLEP 

THE  BEST  (3  0  BEYOND  ^  ■ 

_ Current  Progress _ 

•  ADIU  and  IFSSR  development  efforts  (Cont...). 

-  The  current  ARDS  hardware  configuration  utilizes  a  R3  interface 
between  the  ADIU  and  DLT. 

-  A  new  Synchronous  Data  Link  Control  (SDLC)  interface  has  been 
developed  to  resolve  problems  with  utilizing  the  Range  Encryption 
Module  (REM). 

-  The  new  DRS  developed  ADIU  will  only  work  in  the  SDLC  mode 
and  is  not  backwards  compatible  with  the  R3  interface. 

•  The  Navy  does  not  currently  plan  to  transition  to  the  SDLC 
configuration. 

-  The  Eglin  in-house  developed  ADIU  (referred  to  as  the  EDIU)  is 
backward  compatible  with  the  R3  interface  and  will  also  work  with 
the  new  SDLC  interface. 

•  The  Air  Force  and  Army  have  hard  requirements  to  use  the  REM  and 
are  migrating  to  the  SDLC  configuration  as  a  result. 
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ARDS  SLEP 
Current  Progress 


T/BRIN 


•  ADIU  and  IFSSR  development  efforts  (Cont...)- 

-  The  new  DRS  IFSSR  requires  a  new  end  cap  be  incorporated  in  the 
pod  rail  as  well  as  installing  a  new  battery  holder  in  front  of  the  DLT. 

-  The  Eglin  in-house  developed  IFSSR  (referred  to  as  the  EFSSR)  is 
a  form-fit-function  drop  in  replacement  and  does  not  require  the  new 
end  cap  or  the  relocation  of  the  batter  holder. 

-  The  new  Eglin  EDIU  and  EFSSR  hardware  and  software 
development  is  complete. 

•  Qualification  testing  has  been  completed  including  environmental  stress 
screening,  vibration  and  shock,  and  EMI/EMC. 

•  Certified  Manufacturing  has  been  placed  on  contract  to  build  up  100 
production  units  for  both  the  EDIU  and  EFSSR. 

•  Production  deliveries  are  underway. 
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ARDS  SLEP 
Current  Progress 


T/BRIN 


*  Replacement  DC/DC  power  supply. 

-  No  form-fit-function  drop  in  replacement  was  currently  available. 

-  A  new  DC/DC  power  supply  equipment  specification  was  created 
based  on  previous  SCD’s. 

-  A  limited  open  competition  was  conducted  between  previous  power 
supply  providers. 

-  TYBRIN  awarded  a  contract  to  Technipower  LLC  on  1  December  2008. 

-  The  government  owns  the  full  re-procurement  data  rights  to  the  new  design. 

-  Delivery  of  91  production  power  supplies  is  underway  and  will  be  completed 
by  April  2011. 
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ARDS  SLEP 
Current  Progress 


T/BRIN 


•  ARDS  Pod  Cable  Harnesses. 

-  The  government  qualified  a  new  cable  harness  supplier  to 
manufacture  the  current  ARDS  cable  harness  set. 

-  The  government  now  has  two  qualified  sources  to  procure  the 
ARDS  Red,  Black,  and  REM  By-pass  cable  harnesses  from. 

-  The  new  cable  harness  supplier  provides  a  significant  cost 
savings  while  maintaining  superior  quality  workmanship. 

-  Modifications  have  been  incorporated  into  the  REM  By-pass  cable 
harness  to  allow  it  to  be  interchangeable  with  the  R3  configuration 
and  the  SDLC  configuration. 

•  Previously,  the  REM-By-pass  cable  had  to  be  modified  to  work  in  the 
SDLC  configuration  and  once  modified,  could  no  longer  be  used  in  the 
R3  configuration. 
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ARDS  SLEP 
Current  Progress 


T/BRIN 


•  New  ARDS  Pod  Tube  Hanger  Configurations. 

-  The  ARDS  pod  tubes  currently  have  the  1 ,500  hour  AIM-9  forward, 
center  and  aft  hangers  installed. 

•  Poses  significant  problems  when  flown  on  the  F/A-18. 

•  Limited  number  of  flight  hours  before  they  have  to  be  inspected  for  stress 
and  cracks. 

•  Downtime  for  inspection  is  lengthy. 

-  The  Navy  has  decided  to  move  to  the  new  DRS  proprietary  P-5  TCTS 
forward  hanger  and  government  owned  P4RC  center  and  aft  hanger 
configuration. 

•  Allows  significantly  longer  flight  hours  before  inspections  are  required. 

•  Allows  replacement  of  the  hanger  shoe  on  the  forward  hanger  without 
replacing  the  entire  hanger  band  assembly 
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ARDS  SLEP 
Current  Progress 


T/BRIN 


•  New  ARDS  Pod  Tube  Hanger  Configurations 
(Cont...). 

-  The  ARDS  flight  clearance  for  the  new  Navy  hanger 
configuration  has  been  approved,  and  all  hanger  retrofits 
have  been  completed. 

-  The  Army  also  incorporate  the  Navy  hanger  configuration 
since  they  have  to  support  test  operations  with  the  F/A-18 
as  well  as  the  F-1 5,  F-1 6,  and  A-1 0. 

•  Migrated  to  the  hanger  configuration  that  will  support  the  most 
stringent  requirements  they  have  to  meet  -  F/A-18  E/F  wingtip. 

-  The  46th  TW  is  migrating  to  the  P4RC  forward,  center,  and 
aft  hanger  configuration. 
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ARDS  SLEP 
Current  Progress 


T/BRIN 


•  New  ARDS  Pod  Tube  Hanger  Configurations 
(Cont...). 

-  Edwards  AFB  (AFFTC)  will  stay  with  the  1500  hour 
configuration. 

-  New  Nomenclatures  were  established  for  the  hanger 
configurations. 

•  AN/ARQ-52B  (V)17  Modified  (AFFTC  and  UTTR  configuration) 

•  AN/ARQ-52C  (V)17  New  Navy  and  WSMR  Configuration 

•  AN/ARQ-52D  (V)17  New  Eglin  Configuration 

-  SEEK  EAGLE  fleet  wide  flight  clearance  approval  in  process  for 
all  three  configurations  for  Air  Force  F-15,  F-16,  and  A-10 
aircraft. 


28 


ARDS  SLEP 
Issues 


T/BRIN 


•  The  major  issue  in  the  ARDS  SLEP  has  been 
documentation,  documentation,  documentation! 

-  Incomplete  documentation. 

-  Missing  documentation. 

-  Documentation  not  procured. 

•  Too  many  proprietary  parts. 

-  Undocumented  hardware  and  software  changes  to  the  system. 

-  Documentation  not  properly  validated  and  verified. 

-  Configuration  Management  and  the  documentation  package  was 
the  responsibility  of  the  SPO  at  Eglin  that  procured  the  ARDS 
hardware  up  until  late  2002. 

-  Responsibility  for  CM  and  all  the  documentation  was  transferred  to 
the  Tri-Service  SMO  in  a  formal  transfer  agreement. 
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ARDS  SLEP 
Issues 


T/BRIN 


•  In  preparation  for  the  ARDS  SLEP,  the  T&E  ranges  realized  how 
poor  the  documentation  package  transferred  from  the  acquisition 
SPO  was. 

•  System  performance  specifications  in  general  and  descriptions  of 
how  the  DLT  (network  interfaces)  and  ADIU  operated  and 
interfaced  were  virtually  non-existent  (two  key  components  of  the 
ARDS  hardware  suite). 

-  No  documentation  had  been  procured  in  many  cases. 

-  Documentation  procured  had  been  lost  and  was  no  longer  available. 

-  Many  undocumented  changes  (from  an  ECP  standpoint)  had  been  made  to 
the  DLT  and  ADIU  software. 

•  These  components  became  proprietary  to  the  OEM. 

-  Source  Control  Drawings  for  proprietary  components  had  not  been  procured 
in  in  an  effort  to  cut  costs  in  the  original  acquisition. 

-  Existing  SCD’s  used  in  the  SLEP  were  found  to  have  glaring  errors  that 
should  have  been  discovered  in  the  initial  validation/verification  process. 
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ARDS  SLEP 
Lessons  Learned 


T/BRIN 


•  The  original  FSED  &  LRIP  ARDS  development  produced  a  complete 
build  to  print  Level  III  drawing  package. 

-  All  software  source  code  was  available. 

-  All  hardware  drawings  were  available. 

-  All  system  specifications  were  current  and  accurate. 

-  Allowed  for  open  competition  for  the  full  rate  production  hardware. 

•  By  the  time  Full  Rate  Production  was  completed,  approximately 
50%  of  the  documentation  package  was  no  longer  valid. 

-  Obsolete  components  encountered  during  production  were  engineered 
around  without  proper  documentation. 

-  Enhancements  added  toward  the  end  of  the  production  cycle  (Option  II  on 
the  contract)  were  incorporated  with  no  documentation  procured. 

-  What  started  as  a  100%  government  owned  hardware  and  software  system 
became  a  system  where  all  the  key  components  were  proprietary. 
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ARDS  SLEP 
Lessons  Learned 


T/BRIN 


•  When  originally  procured  and  fielded,  no  thought  was  given  to 
having  to  potentially  sustain  the  system  beyond  its  projected  life 
expectancy. 

•  Emphasis  was  on  buying  more  hardware  and  less 
documentation. 

•  Maintaining  and  properly  archiving  documentation  from  the  initial 
development  (FSED  &  LRIP)  was  not  accomplished. 

-  Not  available  for  ARDS  SLEP  use. 

•  Proper  validation/verification  was  not  completed  on  the 
documentation  package  that  was  maintained. 

•  The  lack  of  proper  documentation  has  resulted  in  a  tremendous 
amount  of  additional  cost  and  time  to  develop  suitable 
replacements  for  key  subsystems  during  the  ARDS  SLEP. 
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ARDS  SLEP 
Conclusions 


T/BRIN 


•  Failure  of  the  EnRAP  program  resulted  in  having  to  keep  the  ARDS 
hardware  suite  operational  long  past  its  projected  life  expectancy. 

-  Planned  operation  after  the  SLEP  is  through  2017. 

•  The  ARDS  hardware  had  a  significant  number  of  obsolete  components. 

•  A  large  effort  has  been  made  to  develop  multiple  sources  of  procurement 
for  many  key  ARDS  components. 

•  The  government  is  working  hard  to  reduce  or  eliminate  proprietary 
components. 

-  Regain  control  and  ownership  of  the  hardware  and  software. 

-  Allow  for  lower  cost  and  quicker  turnaround  in  future  enhancements  and  obsolete 
component  replacement  efforts. 

•  The  ARDS  documentation  package  was  very  incomplete  complicating  the 
ARDS  SLEP  greatly. 

•  Future  CTEIP  programs  should  focus  more  on  ensuring  the  proper 
documentation  is  procured  and  reduce  the  amount  of  proprietary 
components. 
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Megatrends/Imperatives 


Better  Acq:  WSARA  and  the  new  DT&E  Office 


-  Acquisition  Reform  is  still  front  burner  issue 

-  Usual  suspects 

Budget  Reality:  SecDef  Efficiency  Initiatives 

-  Overview 

-  Implications  for  T&E 

-  DOE,  Reliability,  M&S,  IT . 

Imperatives 

-  Current 

-  Future 

-  Cyber 
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hr§  What's  Wrong  with  Acquisition? 


Cost 


THE  USUAL  (?)  SUSPECTS 


Over  Budget 

-  GAO:  96  MDAPs,  $300B  over  initial  estimates 

Schedule 

Late  to  Need 

-  Getting  capability  to  the  user  to  meet  urgent  needs 

Performance 

Programs  failing  Operational  Test 

-  Suitability  issues 

-  Late  discovery  of  failure  modes 

-  Performance  shortfalls 

-  Interoperability 


DISTRIBUTION  STATEMENT  A  -  Cleared  for  public  release 
by  OSR  on  Sep  02  2010  -  SR  case  number  10-S-3203 


UNCLASSIFIED 


6 


Sez  Who? 


Approximately  50%  of  programs  completing 
IOT&E  since  2000  have  been  assessed  as  not 
operationally  effective  and/or  suitable.  ” 

2008  DSB  Report 


.  .beginning  production  before  successfully  demonstrating 
that  the  weapon  system  will  work  as  intended  increases  the 
potential  for  discovering  costly  design  changes. .  .and  usually 
requires  substantial  modification  costs  at  a  later  time.  ” 

2008  GAO  Assessments  of  Selected  Weapon  Programs 
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Bottom  Line 


DoD  Systems  take  too  long  to  field,  cost  too  much  and  don’t 

perform  as  required 
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enter  the  Weapons  System 
Acquisition  Reform  Act 


Purpose  -  Eliminate  waste  and  inefficiency  in 
defense  projects 

Why  -  President  noted  that  the  wasteful 
spending  stems  from: 

-  Out  of  the  ordinary  requirements 

-  No-bid  contracts 

-  Lack  of  oversight 

Concern  -  Schedule  delays  and  cost 
overruns 

How  -  Strengthen  oversight  and 
accountability 

-  Appoint  officials  to  closely  monitor  and  control 
costs 

-  New  offices  of  SE  and  DT&E 

-  Greater  focus  on  testing  new  weapons 

-  Ensure  technologies  are  mature 

-  Ensure  programs  are  started  right 

-  Improve  competition 

-  End  conflicts  of  interest 
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123  STAT.  1710 


PUBLIC  LAW  111-23— MAY  22.  2009 


1 3 1  Section  2366a<  a*  4 1  of  such  title  is  amended  by  inserting 
“,  with  the  concurrence  of  the  Director  of  Cost  Assessment 
and  Program  Evaluation,”  after  “has  been  submitted". 

(41  Section  2366b(.aKlKC)  of  such  title  is  amended  by 
inserting  with  the  concurrence  of  the  Director  of  Cost  Assess¬ 
ment  and  Program  Evaluation,”  after  “have  been  developed 
to  execute". 

(51  Subparagraph  (A)  of  section  2434(b)!  1 1  of  such  title 
is  amended  to  read  as  follows: 

*<  A)  be  prepared  or  approved  by  the  Director  of  Cost  Assessment 
and  Program  Evaluation;  and”. 

(6)  Section  2445c(fK3)  of  such  title  is  amended  by  striking 
"are  reasonable”  and  inserting  “have  been  determined,  with 
the  concurrence  of  the  Director  of  Cost  Assessment  and  Pro¬ 
gram  Evaluation,  to  be  reasonable". 

(e  i  Report  on  Monitoring  of  Operating  and  Support  Costs 
for  Major  Defense  Acquisition  Programs.— 

(li  Report  to  secretary  of  defense. — Not  later  than 
one  year  after  the  date  of  the  enactment  of  this  Act,  the 
Director  of  Cost  Assessment  and  Program  Evaluation  under 
section  139c  of  title  10  United  States  Code  (as  added  by  sub¬ 
section  la)),  shall  review  existing  systems  and  methods  of  the 
Department  of  Defense  for  tracking  and  assessing  operating 
and  support  costs  on  major  defense  acquisition  programs  and 
submit  to  the  Secretary  of  Defense  a  report  on  the  finding 
and  recommendations  of  the  Director  as  a  result  of  the  review, 
including*  an  assessment  by  the  Director  of  the  feasibility  and 
advisability  of  establishing  baselines  for  operating  and  support 
Code. 

Jnsmittai.  to  congress. — NotTHii^han  30  days 
rreceiving  the  report  required  by  paragraph 
^retary  shall  transmit  the  report  to  the  congressional"* 
committees,  together  with  any  comments  on  the  report 1 
Secretary  considers  appropriate. 

SEC.  102.  DIRECTORS  OF  DEVELOPMENTAL  TEST  AND  EVALUATION 
AND  SYSTEMS  ENGINEERING. 

(a)  In  General.— 

( 1 1  Establishment  of  positions. — Chapter  4  of  title  10, 
United  States  Code,  as  amended  by  section  101!  al  of  this  Act, 
is  further  amended  by  inserting  after  section  139c  the  following 
new  section: 

“5 139cL  Director  of  Developmental  Test  and  Evaluation; 

Director  of  Systems  Engineering;  joint  guidance 

“(a)  Director  of  Developmental  Test  and  Evaluation. — 

“<1>  Appointment. — There  is  a  Director  of  Developmental 
Test  and  Evaluation,  who  shall  be  appointed  by  the  Secretary 
of  Defense  from  among  individuals  with  an  expertise  in  test 
and  evaluation. 

“<2>  Principal  advisor  for  developmental  test  and 
EVALUATION. — The  Director  shall  be  the  principal  advisor  to 
the  Secretary  of  Defense  and  the  Under  Secretary  of  Defense 
for  Acouisition,  Technology,  and  Logistics  on  developmental 
test  and  evaluation  in  the  Department  of  Defense. 

“(3)  Supervision. — The  Director  shall  be  subject  to  the  ^ 
supervision  of  the  Under  Secretary  of  Defense  for  Acquisition 
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WSARA  and  DT&E 


The  DDT&E  is  the  principal  advisor  to  the  Secretary  of  Defense  and  the 
Under  Secretary  of  Defense  for  Acquisition,  Technology  and  Logistics 
on  developmental  test  and  evaluation  in  the  DoD 


Responsibilities: 

-  Program  Oversight 

-  Policy  and  Guidance 

-  T&E  Strategy  (TES)  /  TEMP 

-  Acq  DT&E  workforce 

-  Component  T&E  Capability 

-  Annual  Report  to  Congress 


DT&E  in  Title  10,  USC,  Section  139d 

WSARA  signed  May  22,  2009 
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D.DT&E  Mission 


Improve  acquisition 
outcomes  by . 


Supporting: 

•  Acquisition  programs  (planning,  advocacy) 

•  DT&E  workforce  and  community  (advocacy) 

-  Capability  and  competencies 

-  Advancing  “state-of-the-practice” 

-  Policy  development 

•  Decision  Makers 

-  Performance  assessment 

•  Warfighters 


and  minimize  Discovery  in  IOT&E 
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Megatrends/Imperatives 


*  Better  Acq:  WSARA  and  the  new  DT&E  Office 

-  Acquisition  Reform  is  still  front  burner  issue 

-  Usual  suspects 

*  Budget  Reality:  SecDef  Efficiency  Initiatives 

-  Overview 

-  Implications  for  T&E 

-  DOE,  Reliability,  M&S,  IT . 

*  Imperatives 

-  Current 

-  Future 

-  Cyber 
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DoD  Budget  Realities 


•  Although  the  U.S.  faces  significant  economic  challenges  and  growing  budget  deficits,  Defense  base 
funding  must  have  real  growth  to  sustain  force  structure  and  needed  modernization 

-  Fighting  Two  Wars 

-  Confronting  ongoing  terrorist  threats  around  the  globe 

-  Facing  major  powers  investing  heavily  in  their  military 

•  Sustaining  current  force  structure  and  needed  modernization  requires  2-3%  real  growth 

•  The  current  and  planned  base  defense  budget  has  steady,  but  modest  growth  of  1  %  per  year 

•  To  make  up  the  difference  and  preclude  reductions  in  needed  military  capability,  the  difference  of 
1-2%  a  year  will  be  made  up  elsewhere  in  DoD 

•  In  May,  SecDef  began  a  hard,  unsparing  look  at  how  DoD  is  staffed,  organized,  and  operated 


“...in  May  I  called  on  the  Pentagon  to  take  a  hard  and  unsparing  look  at  how  the 
department  is  staffed,  organized  and  operated.  I  concluded  that  our  headquarters  and 
support  bureaucracies,  military  and  civilian  alike,  have  swelled  to  cumbersome  and 
top-heavy  proportions,  grown  over-reliant  on  contractors  and  grown  accustomed  to 
operating  with  little  consideration  to  cost.”  ....Secretary  of  Defense  Robert  M.  Gates 
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enter  the  SecDef  Plan  for 
Efficiency 


•  Target  Affordability  and  Cost 
Growth 

•  Incentivize  Productivity  & 
Innovation  in  Industry 

•  Promote  Real  Competition 

•  Improve  Tradecraft  in  Services 
Acquisition 

•  Reduce  Non-Productive 
Processes  and  Bureaucracy 


(L)  Secretary  of  Defense  Robert  M.  Gates 
(R)  USD  AT&L  Dr  Ashton  B.  Carter 


“Consumers  are  accustomed  to  getting  more  for  their  money  -  a  more  powerful 
computer,  wider  functionality  in  mobile  phones  -  every  year.  When  it  comes  to  the 
defense  sector,  however,  the  taxpayers  had  to  spend  significantly  more  in  order  to  get 
more.  We  need  to  reverse  this  trend.”  ....Secretary  of  Defense  Robert  M.  Gates 
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Why  Test? 


Iterate/Mature  the  Design 
Failure  Mode  Discovery 


Material  Developer 


Inform  Acquisition  Decisions  ^  Decision  Authority  ^ 


•  Confirm  Performance 

•  Safety 

•  Capabilities  and  Limitations 

^  Warfighter  ^ 

‘Testing  is  the  Conscience  of  Acquisition” 
Wiiliam  J.  Perry  -  former  SecDef 
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Knowledge  vs  Cost 
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Challenges  to  doing  good? 


1 .  “Testers  like  to  test” 

-  Who  requires,  who  pays? 

2.  “A  dollar  spent  on  test  is  a 
dollar  spent  on  bad  news” 

-  Incentives  matter 


3.  “Testing  is  driving  up 
our  costs” 

-  Now  vs.  later? 


4.  “We  can’t  afford  it  ” 

-  See  #3 
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and  how  can  T&E  help? 


•  Enterprise  Perspective 

-  Acquisition  Savings 

-  Mature  Systems 

-  Reliability 

-  Early  discovery 

-  Adequate  testing  (early) 

•  Test  Community  Perspective 

-  Recognize  our  role 

-  Manage  our  appetite 

-  Support  the  risk-based  level  of 

information  needed 

-  Do  our  job  more  efficiently 

•  T&E  Cost 

-  Too  much 

-  Bad  news 

-  Late  T&E  Requirements 

•  T&E  Savings 

-  STED  (e  g.,  DOE) 

-  Distributed 

-  CRIS 

-  Capital  Utilization 

-  Integrated  Test 
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Megatrends/Imperatives 


*  Better  Acq:  WSARA  and  the  new  DT&E  Office 

-  Acquisition  Reform  is  still  front  burner  issue 

-  Usual  suspects 

*  Budget  Reality:  SecDef  Efficiency  Initiatives 

-  Overview 

-  Implications  for  T&E 

-  DOE,  Reliability,  M&S,  IT . 

*  Imperatives 

-  Current 

-  Future 

-  Cyber 
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DT&E  Challenges/Imperatives 


•  Support  Acquisition  (WSARA) 

-  Robust,  efficient,  risk-based  T&E 

-  Early  engagement  (Rqmts,  AoA,  RFP,  SS....) 

-  Performance  Assessment  (inform  the  decision  makers) 

•  Support  SecDef  Initiatives  (Efficient  T&E  (doing  more  with  less) 

-  Integrated  Test 

-  DOE 

-  Capital  Utilization 

-  M&S,  ground  testing 

-  Distributed  testing 

•  Reliability 

•  IA  and  10 

•  Cyber 

•  Rapid  Fielding 

•  Workforce  skill  mix 
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Cyber  Warfare 


Computer  Network  Operations 

-  Months,  days,  hours... uSecs 

-  Attribution 

-  Role?  DoD,  Federal,  Civil 
Attack  (CNA) 

-  Precision  strike 

-  Kinetic  effects 
Defense  (CND) 

-  Cyber  missiles 

-  Mission  critical  tasks,  functions 
Exploitation  (ONE) 

-  Intelligence 


“The  best-laid  defenses  on  military  networks  will  matter  little 
unless  our  civilian  critical  infrastructure  is  also  able  to  withstand 

attacks.” . Deputy  Secretary  Bill  Lynn 
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Cyber  Warfare 


What’s  the  role  for  T&E? 

Scope:  Focus  on  CND  and  MDAPs? 

-  Define  cyber  defense  issues  in  network 
environments 

-  What  systems  are  most  vulnerable? 

-  Weapon  systems? 

-  IT  systems? 

-  Rigorous  cyber  defense  testing 

-  Develop  a  cyber  defense  T&E  framework 

-  Institutionalize  cyber  defense  IT 


With  hundreds  of  legacy  and  new  programs  in  development  each 
entering  our  networks,  we  cannot  afford  the  chaos  of  each  one 
individually  planning  or  just  not  testing  for  cyber  defense. 
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OD,DT&E  How-Goz-lt? 


*  Establish  ODDT&E/DDR&E 

*  Organizational  Relationships 

*  Staffing 

*  Director 

*  POM  11 

*  1st  Annual  Report 
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Implications  for  the  Test  and 
Acquisition  Communities 


Enterprise  will  manage  risk 

-  Oversight  and  Accountability 

-  Rapid  vs.  Deliberate  Acquisition 

Visibility 

-  More  emphasis  on  DT 

-  DT&E  voice  at  DAB 

-  Increased  planning  rigor/fidelity 

-  D,DT&E  TEMP  approval 

-  Efficiencies:  DOE,  IT,  M&S . 

Acquisition 

-  Accept  less  risk  at  MS  decisions 

-  More  DT  less  OT? 

-  Confirmation  vs  Discovery 

-  More  informed  decisions 
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DDT&E  is  ... . 


s  Back! 

s  T&E  Community  Advocate! 
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Questions? 


\ 


OFFICE  OF  THE  UNDER  SECRETARY  OF 
DEFENSE  FOR  ACQUISITION, 
TECHNOLOGY  AND  LOGISTICS 


Developmental  Test  &  Evaluation 

3090  Defense  Pentagon 
Room  3B941 

Washington,  DC  20301-3090 

Email:  ddre-dte@osd.mil 

www.acq.osd.mil/dte 


The  right  information,  to  the  right  decision  maker,  at  the  right  time,  for 

better  decisions 
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Back-Up 
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SecDef  Efficiency  Objectives 


•  Deliver  the  warfighting  capability  we  need  for  the  dollars  we  have 

•  Get  better  buying  power  for  warfighter  and  taxpayer 

•  Restore  affordability  to  defense  goods  and  services 

•  Improve  defense  industry  productivity 

•  Remove  government  impediments  to  leanness 

•  Avoid  program  turbulence 

•  Maintain  a  vibrant  and  financially  healthy  defense  industry 
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T&E  Challenges 


Rapid  Fielding 

-  Safety 

-  Caps  and  Lims 

Emerging  Technologies 

How/where  to  test? 

-  Hypersonics 

-  Autonomous  systems 

-  Weaponized  unmanned  systems 

-  Net-enabled  weapons 

Range  Encroachment 

-  OCS  exploration/drilling  ? 

-  Spectrum? 

-  Wind  generators...  !!!!! 
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T&E  Challenges  (continued) 


•  Complex  Systems 

-  System  of  Systems 

-  Interdependent  systems? 

-  Data  fusion 

-  S/W  intensive  systems 


•  Balancing  Adequacy  vs 
Speed  to  Field,  Cost . 

-  DOE? 


Sensar 

Grid 


Re  con/  ^ 
Surveillance 


Mat  Enable 
^  QISR 


■Weapon 
Asset  Mixes 


TracK/llIumlnate 


Joint  Terminal 
Attack  Controllers 


Non-Network  Enabled  Weapons 
(materiel) 


An  borne  C2 


3PS 


Dynamic 

Targeting 

C21SR 


TTP  Options 
imn-msteric.i) 


Joint  Force  Command 


Moving 

Targets 


-  How  much  is  enough?  Risk  management 

-  How  much  M&S?  LVC? 

-  Other  tools 
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T&E  Challenges  (continued) 


•  Reliability 

-  50%  of  MDAPs  are  failing  OT  (Suitability) 

-  DOT&E  imperative  -  RAM  growth  testing 

•  Rigor  -  Realistic  Environments? 

-  Stressing  countermeasures  (GPS  jamming), 

clutter . Operationally  relevant  scenarios 

-  Threat  representations 

•  End-to-End  testing 

-  Mission  Context 
-  Mission  threads 

-  Interoperability  and  IA 
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Encourage  Efficiency 


ADOPTING  “SHOULD-COST”  AND  “WILL-COST”  MANAGEMENT:  Use  historically  informed  independent  cost 
estimation  (“will-cost”  estimates)  to  inform  managing  of  programs  to  cost  objectives  (“should-cost”  estimates). 

STRENGTHENING  THE  ACQUISITION  WORKFORCE:  Achieve  SECDEF  goal  of  adding  to  government  acquisition 
workforce  with  increased  skill  levels.  Leverage  unique  qualities  of  non-profit  FFRDCs  and  UARCs  to  augment  acquisition 
workforce  capability. 

IMPROVING  AUDITS:  Improve  consistency  and  quality  of  government  audits,  and  focus  them  on  value-added  content. 

MANDATING  AFFORDABILITY  AS  A  REQUIREMENT:  In  new  programs  such  as  the  SSBN-X  nuclear  missile 
submarine,  the  Presidential  Helicopter,  the  Ground  Combat  Vehicle,  and  the  Air  Force/Navy  Long  Range  Strike  Family  of 
Systems,  cost  considerations  must  shape  requirements  and  design. 

STABILIZING  PRODUCTION  RATES:  To  ensure  more  programs  are  in  stable,  economically  favorable  rates  of  production 
and  avoid  cost  escalation,  program  managers  may  not  adjust  production  rates  downward  without  head  of  component  authority. 

ELIMINATING  REDUNDANCY  WITHIN  WARFIGHTING  PORTFOLIOS:  Emulate  the  Army’s  Precision  Fires 
Capability  Portfolio  approach  to  identify  where  multiple  programs  are  pursuing  similar  objectives. 

ESTABLISHING  SENIOR  MANAGERS  FOR  PROCUREMENT  OF  SERVICES:  Follow  the  Air  Force  lead  in 
establishing  a  Program  Executive  Officer  for  services  in  each  DOD  component  to  focus  on  improving  policy  and  practice  in  this 
high-dollar-value  area. 

PROTECTING  THE  TECHNOLOGY  BASE:  Protect  the  future  by  sustaining  investment  while  focusing  on  high  value- 
added  work. 
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DoD  HQ  Testing* 


r  \ 

Secretary  of  Defense 

V _ _ ) 


Policy,  guidance,  and  oversight 
to  ensure  OT&E  is  adequate  to 
confirm  operational  effectiveness 
and  suitability  in  combat  use 


Planning  and  assessment  of  the 
adequacy  of  the  Major  Range 
and  Test  Facility  Base  to  provide 
testing  in  support  of  defense 
acquisition 


Policy,  guidance,  and  oversight 
to  ensure  DT&E  is  adequate  to 
support  program  development 
and  assess  system  performance 
for  decision  authority 

DT&E  is  approximately  80%  of 
*Chart  does  not  reflect  entire  OSD  Organization  the  overall  program  T&E  effort 


\ 

Director,  Developmental 
Test  &  Evaluation 
(Mr.  Greer) 
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DDT&E  in  DoD:  Engage  Early 


$1  invested  here 


TEMP 


CCd/s\ 


(Program 

initiation) 


SoTutV  \  Tfjjhnologv 
Anaty^\Deve,°Pment 

Materiel 

■  Development 

Decision  _ 

vPr e-Systems  Acquisitmny/\y 


TEMP 


cpd/c\ 


TEMP 


IOC 


FQC 


Engineer  trig  and 
Hail  ufacturlng 
Development 

Production  & 
Deployment 

Operations  & 
Support 

/%  Posi  /\Post- 
V'  POR  A  VCDRA 

Oiot&e  0  jg*™ 

Systems  Acquisition 


7\ 


Sustainment 


0 


■Test  Evaluation  Strategy 
(TES) 

■Technology 
Development  Strategy 
(TDS) 

■Identify  emerging  T&E 
capability  requirements 
■Identify  T&E  resources 
■Develop  T&E 
requirements  in  RFP 
■Support  AoA  technical 
analysis 


■  Test  and  Evaluation 
Master  Plan  (TEMP) 

■Acquisition  Strategy 

■Support  T&E  Program 
Execution 

■Provide  T&E  results  for 
OIPT/DAB  Reviews 

■CDD  requirements  for 
testability  and  ability  to 
evaluate 

■Support  TRL  Evaluation 

■T&E  requirements  in 
RFP 

■  Report  of  performance 
measures,  metrics  and 
evaluations 


■  Update  TEMP 
■Support  T&E  Program 

Execution 

■Provide  T&E  results  for 
OIPT/DAB  Reviews 
■Support  PDR/CDR  and 
all  technical  reviews 
■CPD  requirements  for 
testability  and  ability  to 
evaluate 

■Support  TRL  Evaluation 
■T&E  requirements  in 
RFP 

■  Discovery  and 
deficiencies 

■  Report  of  performance 
measures,  metrics  and 
evaluations 


■Update  TEMP 
■Support  T&E  Program 
Execution 

■Characterize  system 
capabilities  and 
limitations 

■Provide  T&E  results  for 
OIPT/DAB  Reviews 
■AOTR/OTRR 
■Support  training  for 
IOT&E 

■  Report  of  performance 
measures,  metrics  and 
evaluations 


■  Update  TEMP  for 
Follow-on  DT&E  and 
OT&E 

■Verification  of 
corrections  for 
deficiencies 

■Develop  T&E  programs 
to  support  upgrades, 
modifications,  and 
increments 

■Support  T&E  Program 
Execution 

■  Report  of  performance 
measures,  metrics  and 
evaluations 
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EjjJ  DDT&E  in  DoD:  What  and  When 


$1  invested  here 


TES 


TEMP 


CCD 


TEMP 


(Program 

initiation) 


Sol  \ 

Anaiyi»w  ’ 

Technology 

1  Development 

w  Materiel 

>  Development! 

CPD 


Engineering  ar 
Manufacti 
De\ 


Saves  $XXX  here 


^Pre-Systern^Xcq^igitiQny^^ 


o 


FOC 


Operations  & 
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Systems  Acquisition 


/\  Sustainment 


A  few  DDT&E  observations: 

*  Lack  of  DT&E  expertise  during  program  formulation 

*  DT&E  program  planning  and  resourcing  not  adequate 

*  System  immaturity  at  MS  C  or  at  OTRR 

*  Inadequate  reliability  growth  programs 


DISTRIBUTION  STATEMENT  A  -  Cleared  for  public  release 
by  OSR  on  Sep  02  2010  -  SR  case  number  10-S-3203 


UNCLASSIFIED 


35 


Where  are  We  Going? 


Integrated  Test/Training 


Platform-Based 


System  of  Systems 


Threat-Based 


Complex  Capabilities 


Contract  Compliance 


Mission  Context 


Interoperability 


Joint  Mission  Thread 


Deliberate 


Serial  Testing 


Rapid/Responsiveness 


Our  T&E  process  needs  to  evolve  to  support  faster  product  cycles, 
more  adaptable  products  and  address  challenges 
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Guidance  Roadmap 


Target  Affordability  and  Control  Cost  Growth 

Mandate  affordability  as  a  requirement 

-  At  Milestone  A  set  affordability  target  as  a  Key 
Performance  Parameter 

-  At  Milestone  B  establish  engineering  trades  showing 
how  each  key  design  feature  affects  the  target  cost 

Drive  productivity  growth  through  Will  Cost/Should  Cost 
management 

Eliminate  redundancy  within  warfighter  portfolios 
Make  production  rates  economical  and  hold  them  stable 
Set  shorter  program  timelines  and  manage  to  them 

Incentivize  Productivity  &  Innovation  in  Industry 

Reward  contractors  for  successful  supply  chain  and  indirect 
expense  management 

Increase  the  use  of  FPIF  contract  type  where  appropriate 
using  a  50/50  share  line  and  120  percent  ceiling  as  a  point  of 
departure 

Adjust  progress  payments  to  incentivize  performance 
Extend  the  Navy’s  Preferred  Supplier  Program  to  a  DoD-wide 
pilot 

Reinvigorate  industry’s  independent  research  and 
development  and  protect  the  defense  technology  base 

Promote  Real  Competition 

Present  a  competitive  strategy  at  each  program  milestone 
Remove  obstacles  to  competition 

-  Allow  reasonable  time  to  bid 

-  Require  non-certified  cost  and  pricing  data  on  single 
offers 

-  Require  open  system  architectures  and  set  rules  for 
acquisition  of  technical  data  rights 

Increase  dynamic  small  business  role  in  defense 
marketplace  competition 


Improve  Tradecraft  in  Services  Acquisition 

-  Create  a  senior  manager  for  acquisition  of  services  in  each  component, 
following  the  Air  Force’s  example 

-  Adopt  uniform  taxonomy  for  different  types  of  services 

-  Address  causes  of  poor  tradecraft  in  services  acquisition 

-  Assist  users  of  services  to  define  requirements  and  prevent 
creep  via  requirements  templates 

-  Assist  users  of  services  to  conduct  market  research  to  support 
competition  and  pricing 

-  Enhance  competition  by  requiring  more  frequent  re-compete  of 
knowledge-based  services 

-  Limit  the  use  of  time  and  materials  and  award  fee  contracts  for 
services 

-  Require  that  services  contracts  exceeding  $1 B  contain  cost 
efficiency  objectives 

-  Increase  small  business  participation  in  providing  services 

Reduce  Non-Productive  Processes  and  Bureaucracy 

-  Reduce  the  number  of  OSD-level  reviews  to  those  necessary  to  support 
major  investment  decisions  or  to  uncover  and  respond  to  significant 
program  execution  issues 

-  Eliminate  low-value-added  statutory  processes 

-  Reduce  by  half  the  volume  and  cost  of  internal  and  congressional 
reports 

-  Reduce  non-value-added  overhead  imposed  on  industry 

-  Align  DCMA  and  DCAA  processes  to  ensure  work  is  complementary 

-  Increase  use  of  Forward  Pricing  Rate  Recommendations  (FPRRs)  to 
reduce  administrative  costs 
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Providing  Incentives  for  Greater 
Efficiency  in  Industry 


•  LEVERAGING  REAL  COMPETITION:  Avoid  directed  buys  and  other  substitutes  for  real  competition.  Use  technical  data  packages  and  open 
systems  architectures  to  support  a  continuous  competitive  environment. 


•  USING  PROPER  CONTRACT  TYPE  FOR  DEVELOPMENT  AND  PROCUREMENT:  Phase  out  award-fee  contracts  and  favor  fixed-price  or  cost-type 
incentive  contracts  in  which  government  and  industry  share  equally  in  overruns  and  underruns,  and  overruns  have  analytically-based  caps. 

Use  cost-reimbursement  contracts  only  when  either  government  requirements  or  industry  processes  cannot  be  adequately  specified  to  support 
pricing.  Adjust  sole-source  fixed-price  contracts  over  time  to  reflect  realized  costs.  Work  down  undefinitized  contract  actions.  Seek  authority 
for  multi-year  contracts  where  significant  savings  are  possible. 

•  USING  PROPER  CONTRACT  TYPE  FOR  SERVICES:  Phase  out  Time  and  Material  and  sole-source  ID/IQ  contracts  wherever  possible.  Utilize 
fixed-price  performance-based  contracts  when  requirements  are  firm  and  can  be  measured,  with  payments  tied  to  performance.  Utilize  fixed- 
price  level  of  effort  or  cost-plus-fixed-fee  contracts  (with  profit/fee  tied  to  weighted  guidelines)  when  requirements  are  still  being  defined. 

Award  fees  should  be  used  only  by  exception.  Maximize  the  use  of  multiple-source,  continuously  competitive  contracts. 

•  ALIGNING  POLICY  ON  PROFIT  AND  FEE  TO  CIRCUMSTANCE:  Align  opportunity  to  earn  profits/fees  to  both  value  to  the  taxpayer  and  risk  to  the 
contractor.  Apply  weighted  guidelines  to  profit/fee  levels.  Reward  higher  productivity  with  higher  profits.  Incentivize  investment  in  innovation. 

•  SHARING  THE  BENEFITS  OF  CASH  FLOW:  Ensure  that  taxpayers  receive  adequate  consideration  (price  reductions)  for  improved  cash  flows. 
Progress  payments  must  reflect  performance  but  can  be  increased  above  customary  levels  in  return  for  consideration  by  the  contractor. 

Reduce  over  time  the  gap  between  proposed  and  actual  rates  in  forward  price  rate  agreements. 

•  TARGETING  NON-VALUE-ADDED  COSTS:  Identify  and  eliminate  non-value-added  overhead  and  G&A  charged  to  contracts.  Limit  fees  for 
subcontractor  management  to  reflect  actual  value  provided  (risk  assumed  by  prime  and  continuous  subcontractor  risk  reduction).  Limit  B&P 
allowable  costs  in  sole  source  contracts  and  encourage  effective  use  of  IRAD. 

•  INVOLVING  DYNAMIC  SMALL  BUSINESS  IN  DEFENSE:  When  establishing  multiple  award  contracts  for  services,  make  every  effort  to  provide 
for  small  business  participation.  If  at  least  two  small  businesses  are  deemed  capable  of  performing  on  such  a  contract,  consider  setting  aside 
that  work  for  competition  among  them. 

•  REWARDING  EXCELLENT  SUPPLIERS:  Emulate  the  Navy’s  pilot  program  to  provide  special  benefits  to  consistently  excellent  industrial 
performers. 
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National  Defense  Industrial  Association 
27th  Annual  Test  and  Evaluation  Conference 

Tampa,  FL 

T&E  Instrumentation  Infrastructure  -  Maximum  Utilization  of  Available  Resources  Session 

16  March  2011 


Briefed  by: 

Jim  Schwierling 

Lead  Project  Director,  PMITTS,  PEO  STRI 
Telephone:  256-876-3451  DSN:  746-2451 
Electronic  Mail:  jim.schwierling@us.army.mil 
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commitment  nor  obligation  to  provide 
any  additional  detailed  information  or  an 
agreement  of  sale  on  any  of  the  systems 
or  capabilities  portrayed  during  this 
presentation  that  have  not  been 
authorized  for  release. 
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12350  Research  Parkway 
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Mr.  Mark  C.  Tutten 
Threat  Systems  Mgmt  Office 
ATTN:  SFAE-STRI-PMITTS-S 
Redstone  Arsenal,  AL  35898- 
4761 

(256)  876-9656  x200 
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email: 
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•  MANAGE  TOTAL  LIFE  CYCLE  OF  TARGETS, 
OPERATIONAL  THREAT  VEHICLES,  TARGET  CONTROL 
SYSTEMS  AND  GROUND  RANGE  SYSTEMS  USED  IN 
LIVE  AND  VIRTUAL  TESTING,  AND  TRAINING. 

•  PROVIDE  BEST  VALUE  ACQUISITION,  SUPERIOR  LIFE 
CYCLE  SUSTAINMENT  AND  OPERATION  FOR  THE  U.S. 
ARMY,  DoD,  AND  INTERNATIONAL  CUSTOMER. 

•  EXECUTE  MISSIONS  AS  ASSIGNED  OR  DIRECTED  BY 
PEO  STRI  AND  PM  ITTS. 
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PRIMARY  ACTIVITIES 


Based  on  Customer  Target  1 

Requirements  r 
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•  Aerial  -  Fixed  and  Rotary  Wing 

•  Mobile  Ground  /  Foreign  Materiel 
(both  conventional  and  unconventional) 

-  Foreign  Systems 

-  Surrogates 

•  Virtual  -  Models  and  Simulations 

•  Precision  Targetry  Systems 

•  Auxiliary  /  Ancillary  Equipment 

P 
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WHAT  WE  DO 


MATERIAL  DEVELOPER 


CUSTOMER  SUPPORT 


Fly 


Drive 


Sustain 


AERIAL  TARGETS 


Remote  Piloted  Vehicle  Target 


Turnkey  Operations 
Target  systems  flight 
services  supporting  Army 
and  Tri-service  test  and 
training  and  FMS 
requirements 
Low  Cost 


UAS-T 


Simulate  Aerial  Threats  World-Wide  in  Live  and  Virtual  Domains 


Broadsword  and  Outlaw 

€  ■ 


Broadsword 


.ft* 


Length . 13.7  ft  (4.2  m) 

Wingspan . 17.21ft  (5.2  m) 

Speed . 132  mph 

Average  Basic  Weight.. .350  lbs 
Gross  Weight . 550  lbs 


BroadSword  is 
available  in  carbon 
fiber  or  fiberglass  for 
variable  radar 
signatures 


Payload  is  a  trade 
between  required 
airspeed,  altitude, 

and  flight  duration 


Outlaw 


Length . 8.4  ft  (2.56  m) 

Wingspan. ..13.6  ft  (4.15  m) 

Speed . 125  mph 

Average  Basic  Weight.. .70  lbs 
Gross  Weight...  120  lbs 


AERIAL  TARGETS 


MOBILE  GROUND  TARGETS 


Prasbion  Turyai 
Siynaiu  ra 


Centrally  Manage  and  Execute : 

•  Over  340  assets 

*  Mobile  Ground  Targets  for 
development  and 
operational  testing 

*  Multiple  usage  options: 

*Rent 

•| 

•Buy 

•  Provide  accreditation 
support 


Range  Targetry 

•  Design 

•  Procurement 

•  Fielding 

•  Support 


J  Foreign  Equipment 
□  Surrogate  Equipment 


Threat  Representative  Targets  in  Live  and  Virtual  Domains 


MOBILE  GROUND  TARGETS 


•  Virtu 


nsltionlng  CAD  models  into 
d,  and  radar  frequency 


Target  Generation  Laborati 
simulation  compliant  visual,  i 

simulation  target  models 

rmy  Model  Exchanger.  Distributing  simulation  target  models 

elopers  throughout  the  Army  T&E  community 


Army  Model  Exchange 
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O  Army  Model  Exchange  -  Windows  Internet  Explorer 
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Jg  https://modelexchange.army.mil/ 
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Edit  View  Favorites  Tools  Help 


The  Army  Model  Exchange  is  sponsored  by  the  U.S.  Army  PEO  STRI  Targets  Management  Office  and  the  RDECOM  Virtual  Targets  Center.  It  was  created  to 
promote  model  reuse  for  all  DoD  agencies  involved  in  modeling  and  simulation.  It  is  located  at  the  System  Simulation  &  Development  Directorate  of  the  RDECOM 
AMRDEC. 

Log  In  To  The  Model  Exchange^ 


Army  Model  Exchangejfia#be  accessed  in  two  differ«t  ways,  cKcribed  below. 
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Army  Model  Exchange  Managed  Account  _ 

People  requiring  full  access  to  Army  Model  Exchange  assets  can  apply  for  a  model  exchange  account. 

You  must  register  for  an  account. 

Army  Knowledge  Online  (AKO)  Managed  Account 

Army  Knowledge  Online  (AKO)  account  holders  are  automatically  granted  limited  access  to  the  Army  Model  Exchange's  visualization  models.  Just  log  right  in  using  your 
AKO  account  name\password. 

To  obtain  an  AKO  account,  visit:  https://www.us.army.mil 


and  Downl 


Terms  of  Use  /  Terms  of  Service 


Ava  i  I  at)  le  Models  through  the  Army  Model 

to  facilitate  protection  against  unauthorized  access,  and  to  verify  security^^iedures,  survivability,  and  operational  security.  Motoring  includes,  but  is  not  limited 
to,  active  attacks  by  authorized  DoD  entities  to  test  or  verify  the  security  of  this  system.  During  monitoring,  information  may  be  examined,  recorded,  copied  and 
used  for  authorized  purposes.  All  information,  including  personal  information,  placed  on  or  sent  over  this  system  may  be  monitored.  Use  of  this  DoD  web  site, 
authorized  or  unauthorized,  constitutes  consent  to  monitoring.  Unauthorized  use  of  this  DoD  web  site  may  subject  you  to  criminal  prosecution.  Evidence  of 
unauthorized  use  collected  during  monitoring  may  be  used  for  administrative,  criminal  or  other  adverse  action. 


Use  of  this  system  constitutes  consent  to  monitoring  for  all  lawful  purposes. 


Trusted  sites 


Start  j  £  Pages  -  BlackDart_201 1  . . .  \\  Army  Model  Exchang...  LJ  NDIA  T&E  Presentation  |  fcj  Microsoft  PowerPoint  -  [. . . 
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WHAT  WE  HAVE  DEVELOPED/ 
PURCHASED  RECENTLY 


Precision  Target  Signature 


FIVE  YEAR  FORECAST  TO 
DEVEL  OP/PURCHASE 


Precision  Targets  - 
Mobility 


Fully  Mission  Capable 


Medium  Speed  Aerial  Targets 


Technical  Vehicle  w/crew 


representation 


High  Speed  Aerial  Targets 


Rotary  Wing  Targets 


Common  Control  System 


SUMMARY 


TMO: 

•  ALWAYS  LOOKING  FOR  A  BETTER,  FASTER, 
CHEAPER  PRODUCT  FOR  OUR  CUSTOMERS 

•  RECOGNIZED  LEADER  OF  AERIAL  AND  GROUND 
TARGETS 

•  READY  TO  RESPONSIVELY  AND  RESPONSIBLY 
SUPPORT  T&E  AND  SPECIAL  TRAINING 
REQUIREMENTS 


NEED  INDUSTRY  TO  CONTINUE  PROVIDING 
REASONABLY  PRICED,  STATE  OF  THE  ART 
TECHNOLOGIES  FOR  ADAPTATION  AND 
INCORPORATION  INTO  TARGETRY 


PROVIDING/OPERATING  AERIAL, 
GROUND  &  VIRTUAL  TARGETS 


Raytheon 

Customer  Success  Is  Our  Mission 


2011  NDIA  Test  and  Evaluation 

Conference 

Continuous  Cost  Reduction 
Feeds  Back  Into  Product 

Reliability 


Author:  Jonathan  Nikkei 
Raytheon  Missile  Systems 
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120GM  Dagger™  Introduction 


Raytheon 

Missile  Systems 
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120GM  Dagger™ 


Raytheon 

Missile  Systems 


■  Advanced  Precision  Mortar  Initiative 

-  2009-Present  Urgent  Need  Effort  to  Expedite  Guided  120mm  Mortars  to 
Field 

-  RMS  was  awarded  a  Phase  1  contract 

-  APMI  Phase  2  contract  (sole  source)  was  awarded  to  ATK 

■  Raytheon  120GM  Dagger™  GPS-only  Design  was 
updated  during  APMI  Phase  1  to  include 

-  Standard  Weapon  Interface  Compatibility 

-  SAASM  GPS 

-  Telemetry 

-  Tri-Mode  Fuze  (Standard  M734A1  Mortar  Fuze) 
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Reliability 


Raytheon 

Missile  Systems 


■  Many  definitions,  a  good  definition: 

-  “The  probability  that  a  functional  unit  will  perform  its  required 
functions  for  a  specified  interval  under  stated  conditions." 

■  How  is  reliability  scored/evaluated? 

-  Analytical  Methods  (mostly  pre-CDR) 

■  Our  program  conducted  minimal  effort  here  (quick  turn,  no  time) 

■  Created  fault  trees,  use  of  Built-in-Test 

-  Test  and  Evaluation  (mostly  post-CDR) 

■  Heavy  emphasis  on  component/system  level  repeatability  testing  and  All- 
Up-Round  Flight  Testing 

■  Simple  sequence:  Test  system,  find  problems,  fix  them,  test  again. 

■  In  general,  product  reliability  is  proportional  to 

-  Man-hours  Invested  in  T&E 

-  Number  of  Hardware  Units  Built/Delivered 
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Reliability 


Raytheon 

Missile  Systems 


■  Understanding  and  Achieving  Reliability  in 
Missile/Projectile  Business  can  be  a  Difficult  Problem  Due 
to  Intrinsic  Nature  of  Expendable  Systems 

(not  to  say  it  isn’t  difficult  elsewhere...) 

-  Long  dormant  storage  life  requirements 

-  1 -shot  devices  (squibs) 

-  No/minimal  design  capacity  for  built-in  redundancy 

-  Minimal  information  from  systems  under  test  (sometimes  must 
disturb  system  to  extract  information) 

-  Difficult  environmental  requirements 

-  Shoe-string,  leap-frog  budgets 

-  Tight  schedules  when  money  is  present 
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Complex  Technology  Products 
Reliability  Incentives 
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Missile  Systems 
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Sample  Product  Spectrum 
120GM  Dagger™ 
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UAV’s,  Missiles,  Glide 
Bombs 

(A-A,  A-G,  S-A) 


Missile 

Defense 


Unmanned 

Spaceflight 


Manned 
Spaceflight 


All  Up  Production  Price  (AUPP), 

Product  Complexity 

Location  on  this  curve  largely  dictates  T&E  behavior. 

„  e  We  should  strive  to  move  towards  less  complexity/price! 

Customer  SUCucss  IS  'uui  iviissiuii  is  a  i  eyisiei  cu  u  auei  i  ian\  ui  rvayuieuii  ouni|jaiiy.  1  J  1 


Sources  of  Product  Maturity 


Raytheon 

Missile  Systems 


■  Laboratory  Testing 

-  Use  case  parameter  exploration  with  hardware 

-  Software  parameter  exploration 

-  Functional  testing 

-  Repeatability  testing 

■  Extremely  Boring,  Extremely  Effective! 

■  Simulation 

-  Some  mix  of  real  and  simulated  hardware  and  physics 

-  Performance  optimization 

-  Rapid  software  evolution 

-  Software  parameter  exploration 

■  Field/Flight  Testing 

-  Real  product  hardware  in  tactical  or  near-tactical  environment 
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Raytheon 

Missile  Systems 


Optimal  Mixture  is  Product  Dependent 

■  Optimal  Test  Mixture  Depends  on  Location  in  Product 


120GM 

Dagger™ 


Space 


■  High  Failure  Tolerance/Low  Production  Price 

-  Laboratory  testing  as  necessary 

-  Minimalistic  (low  fidelity)  simulation  necessary  to  mature  software 
algorithms  and  generate  course  performance  estimates 

-  Heavy  weighting  towards  field/flight  testing  with  real  hardware,  as  soon  as 
possible  (10’s  to  100’s  of  flights  per  year) 

■  Low  Failure  Tolerance/High  Production  Price 

-  Heavy  laboratory  testing 

-  Heavy  work  in  low,  medium,  and  high  fidelity  simulations 

-  Field/Flight  test  minimally,  and  only  once  high  confidence  in  success  is 
achieved  (1-10  flights  per  year) 
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Types  of  T&E  -  Pros/Cons 


Laboratory/Simulation  Testing 


PRO  -Usually  Cheaper  than  Flight 
Testing  (both  monetarily  and 
politically) 

•Easy  to  control,  homogenize  and 
selectively  explore  product 
parameter  space 
•Failures  have  minimal  political 
impact 

CON  -Lower  Fidelity  than  Flight  Testing 
•Mountains  of  Data 
•Time  Consuming 
•Inaccuracy  in  Performance 
Estimates  due  to  Modeling  Fidelity 
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Raytheon 

Missile  Systems 


Field/Flight  Testing 


•Highest  Fidelity 
•High  Political  Impact 
•Exposes  Product  Issues  Quickly 
•True  Performance  Estimates 


•High  (Negative)  Political  Impact 
•Expensive 

•Tendency  to  heavily  script  events  due 
to  political  risks 

•Larger  Non-Homogeneous,  Random 
Parameter  Space  that  is  Difficult  to 
Quantify/Measure/Control/Understand 


On  the  “Fire  and  Fix”  Mentality 


Raytheon 

Missile  Systems 


■  Thomas  Edison  vs.  Nicola  Tesla 

-  Tesla  hated  the  experimental,  non-theoretical  methods  Edison  used 

-  Tesla  was  (and  is  still)  revered  for  his  theoretical  prowess 

-  In  the  end,  Tesla  was  not  a  successful  businessman  -  he  was  too 
academic! 

-  Edison  did  not  need  to  fully  understand  the  underlying  physics  to  make 
something  work 

■  When  time  is  short,  and  hardware  is  (relatively)  cheap,  one  can 
resort  to  experimental  methods. 

■  Even  though  it  does  not  sound  as  “smart”  (because  it  is  not!), 
experimental  methods  can  be  (and  have  been  for  us)  a 
legitimate  approach  to  maturing  a  product. 

■  Both  men  and  their  methods  represent  extremes  -  a  mix  of 
laboratory,  simulation,  and  flight  testing  is  best 
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Risk  Aversion 
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■  Why  do  we  fear  failure? 

-  Yields  Negative  Customer  Perception:  “This  Widget  Will  Never  Meet 
Performance/Reliably  Within  a  Schedule  We  Care  About.” 

■  Certainly,  life  is  cozier  if  we  never  fail 

■  Failure  is  often  a  necessary  step  in  maturing  a  product 

-  We  must  increase  our  appetite  to  budget  for  failure,  and  build  failure  into 
(some)  programs... this  is  difficult  to  sell  in  an  era  of  declining  expenditures. 

-  Desire  is  to  work  testing  towards  the  edge  of  the  performance  envelope,  out  of 
the  cushy  nominal  areas,  as  political  landscape  allows.  We  want  to 
understand  where  and  why  a  widget  fails! 

-  Failure-tolerant  programs  are  more  likely  to  be  successful  in  the  end. 

■  Failure  Often  Yields  More  Knowledge  and  Product 

Improvement  than  Success,  because  Engineers  are  Forced  to 

Dig  Deeper 

■  Don’t  Dread  the  Failure  Review  Board  -  Embrace  the 

Opportunity  to  Learn  Something  New 
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Rate  of  Product  Maturity  Increase 


Product  Maturity  Incentives 


Raytheon 
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Example  AUPP  vs.  Flight  Test  Quantities  Raytheon 
Economies  of  Scale  Missile  Systems 


■  Unit  Cost  Reduction  Feeds  Back  Into  Product  Reliability  by 
Allowing  Us  To  Extract  more  Knowledge  from  a  Given 
Budget 

■  Notional  Analysis  -  synthetic  costing/budget  numbers,  not 
real  data 

-  Values  used  are  for  example  purposes  only 

-  Low  Quantity  or  Initial  AUPP:  $19k 

-  Notional  -Logarithmic  Price  Breaks 

-  FYXX  T&E  Materials  Budget:  $800,000 

■  le,  customer  gives  us  $800k  for  flight  testing  this  year.  What  can  we  do 
with  it? 
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AUPP  vs.  Flight  Test  Quantities  (cont) 
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(Synthetic 
Information,  Not 
Real  Costing  Data) 

Example: 
Achieving  50% 
cost  reduction 
more  than 
doubles  our  test 
articles  at  this 
budget  level, 
because  we  hit 
the  next  level  of 
price  break. 

Accelerates  us 
into  regime  of 
finding/fixing 
the  nitty-gritty 
1%  failures! 
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Flight  Test  Quantities  vs.  AUPP 
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Incentive:  Cost  Reduction  Increases  Impact 
of  Price  Breaks  on  Test  Article  Quantities 
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How  Do  We  Minimize  Cost? 
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■  A  Few  Strategies  Employed 
-  Migration  functionality  of  multiple  CCA’s  into  a  single 


CCA 


-  No  wheel  re-invention  -  use  of  proven  COTS  component 
parts 

-  Move  from  milled  to  extruded  or  cast  metal  parts  where 
possible 

-  Reduce  number  of  metal  parts 

-  Phase  in  next  generation  component  parts  (vendor 
produces  a  lower  cost  alternative) 

-  Minimize  Test  Equipment  NRE 

-Automate  assembly  and  test  processes  to  reduce  test 
time 
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Where  We  Are 
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■  Status 

-  Post-APMI  Phase  1,  team  size  was  significantly  reduced 

-  Reliability  improvement  work  has  continued  on  a  shoe-string  budget 

-  An  unconventional  first:  This  program  validated  improvements  in  flight  test  with  re¬ 
used  spent  flight  hardware  (shot  out  of  a  gun,  impacted  the  ground),  in  one  case 
with  3x  re-use  (guidance  electronics  only,  no  structural  components).  Third  HW 
flight  after  problem  fixes  missed  target  by  <1m! 

-  We  have  conducted  many  recent  successful  firing  tests,  with  major  hardware 
components  donated  by  suppliers! 

-  We  wish  to  thank  our  supporters  at  Picatinny  Arsennal,  Yuma  Proving  Ground,  and 
New  Mexico  Tech 

■  120GM  Dagger™ 

-  Extended  Range 

-  High  Accuracy,  Even  in  Moderate  Winds 

-  No  MET  data  required 

-  Tri-mode  Fuze 
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Impact  Video  from  APMI  Shoot-Off 
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File  Type:  TIFF 
Camera  ID:  1 

Camera  Name:  GUN  IMPACT 

Camera  Type:  Color 

Pecord  Pate:  1 0£)0 

Pecord  Mode:  Stop 

Hub  Present:  False 

White  Balance:  User 

Exposure:  700 

Session  ID:  40 

Session  Name:  HGXP 

Serial  Number:  270076 

Orientation:  Upright 

Date:  02/12/10 

Time:  01 :  40:33 

Frame:  -845 

Minutes:  0 

Seconds: 0 

^)S: -8451 81 

I  rig  Days:  42 

Irig  Hours:  20 

Irig  Minutes:  33 

Irig  Seconds:  52 

Irig  |iS:  416000 

iPlG  Offset:  Start 

IPIG  Phase  Shift:  0 

Time  Source:  GPS 

Time  Lock:  True 
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Flight  Test  Results 

June  2010  Reliability  Improvements 
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Fired 
with  2.5 
deg 

ballistic 

azimuth 


from 

target! 


Energy  On  Target! 
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Conclusions  -  Necessary  Mindsets 
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Missile  Systems 


■  Drive  Down  Cost  Early  in  the  Design  Cycle  to 

Reap  the  Rewards  of  Economies  of  Scale 

■  Change  is  necessary  to  mature  a  product 

■  Challenge  Consensus 

-The  fact  that  10  people  believe  something  and  agree 
with  each  other  does  not  make  them  correct! 

-Just  because  something  has  always  been  done  a 
certain  way,  does  not  imply  it  is  correct! 

-  Be  the  outlier... ask  the  question,  even  if  you  think  you 
are  going  to  get  laughed  out  of  the  room! 
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Conclusions  (cont) 
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■  Abnormal/variable  product  behavior  under 
constant  conditions,  even  if  it  does  not  result  in  a 
high  level  product  failure  is  not  ok! 

-  Don’t  be  the  one  who  says:  “Oh  it’s  ok... it  just  does  that 
sometimes...” 
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Contact  Info 
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■  Jonathan  Nikkei 

-  Title:  Sr.  Systems  Engineer 

-  Company:  Raytheon  Missile  Systems 

-  Phone:  520-545-9421 

-  Email:  nikkel@raytheon.com 
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OKA 

A  Combat  Support  Agency 


"The  information  provided  in  this  briefing  is  for 
general  information  purposes  only.  It  does  not 
constitute  a  commitment  on  behalf  of  the  United 
States  Government  to  provide  any  of  the 
capabilities,  systems  or  equipment  presented  and  in 
no  way  obligates  the  United  States  Government  to 
enter  into  any  future  agreements  with  regard  to  the 
same.  The  information  presented  may  not  be 
disseminated  without  the  express  consent  of  the 

United  States  Government." 
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Purpose 


A  Combat  Support  Agency 


Provide  an  overview  of  the 
policies,  processes  and 
procedures  for  assessing 
compliance  with  the  Net-Ready 
Key  Performance  Parameter 


Goal:  Establish  a  measurable,  testable,  and 
operationally  relevant  approach  to  Joint  interoperability  (IOP) 
engineering,  test,  evaluation  &  certification  (TE&C) 


A  Combat  Support  Agency 


Policy  Overview 


► 


JS  -  Interoperability  Certification 


DOT&E  -  Operational  Test  Reports 


DD  4630.5 


“IT  and  NSS  interoperability 
shall  be  verified  early, 
and  with  sufficient  frequency 
throughout  a  system’s  life  ...” 


Title  10 

United  States  Code  (USC 


Section  2223 

IT:  Additional  Responsibilities  of  DoD  CIO 
"Ensure  the  interoperability  of 
Information  Technology  and 
National  Security  Systems 
throughout  the  DoD." 


“All  IT  and  NSS  must  be 
evaluated  and  certified  for 
Joint  interoperability  by  DISA 
(JITC).” 


I  4630.8 

“All  fT  and  NSS  ...  must  be 
tested  for  interoperability 
before  fielding  ...  and 
certified  by  DISA  (JITC).” 


3170.01G 


Establishes  JCIDS 
w/  NR-KPP  for 
CDD and  CPD 


n^j  5nnn  *t*r\e* 

“For  IT  systems,  including  NSS, 
..'JITC  shall  provide  system 
interoperability  test  certification 
memoranda  ...  throughout  the 
system  life-cycle  and 
regardless  of  ACAT” 


United  States  Code  (USC) 


Section  139:  “The  Director  [OT&E]  shall  prescribe, 
policies  and  procedures  for  the  conduct  of 
OT&E  in  the  DoD. ..and  report  test  results 
to  Congress...” 

II.  Section  2399:  OT&E  must  be  adequate,  and 
determine  operational  effectiveness  and 
suitability 


IT  &  EVALUATION 


EVALUATION 


fJT&E)  PROGRAM 


“A  JT&E  is  OT&E  that  brings 
Military  Departments  together 
to  assess  Service  interoperability 
in  joint  operations.” 


OTA  MISSION 

“JITC  shall  perform  the  OTA  mission.. 
The  Commander,  JITC,  will  report 
directly  to  the  Director,  DISA, 
on  OT&E  matters.” 
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HIQf  Joint  Interoperability  Test 


A  Combat  Support  Agency 


Certification  Overview 


NR-KPP  Elements: 

Compliant  Solution 
Architectures 

Net-Centric  Data  and 
Services  Strategy 

GIG  Technical  Guidance 

Information  Assurance 

Supportability 


•  The  NR-KPP  elements  define  the  areas  JITC  evaluates  for 
interoperability  certification 

•  JITC  uses  data  collected  during  DT,  OT,  demonstrations, 
exercises,  or  other  reliable  sources  for  interoperability 
evaluations 


Success  =  Minimizing  separate  interoperability  testing  by  leveraging  DT/OT 


A  Combat  Support  Agency 


Joint  Interoperability 
Certification  Process 


Joint  Staff  J-6 

Interoperability  & 
Supportability 
Certification 
Documents: 


CDD,  CPD,  ISP,  ISP 
Annex  and  TISP 


JITC  Test  &  Certification 


j  Developmental  and  Operational 
:  Test  &  Evaluation 


Joint  Interoperability  Test 
Certification 


Expires  after  4  years,  or  upon  changes  affecting 
interoperability  (system  or  environment) 


NOTE:  Interoperability 
changes  require 
reentering  process  at 
appropriate  point: 


S  Requirements  updates 
S  J-6  l&S  Certification 
S  JITC  Test  &  Certification 
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NR-KPP  Statement 


Net-Ready:  The 
capability,  system,  and/or 
service  must  support  Net- 
Centric  military 
operations.  The 
capability,  system,  and/or 
service  must  be  able  to 
enter  and  be  managed  in 
the  network,  and 
exchange  data  in  a 
secure  manner  to 
enhance  mission 
effectiveness.  The 
capability,  system,  and/or 
service  must 
continuously  provide 
survivable,  interoperable, 
secure,  and  operationally 
effective  information 
exchanges  to  enable  a 
Net-Centric  military 
capability. 


Threshold 


The  capability,  system,  and/or  service  must  fully 
support  execution  of  joint  critical  operational 
activities  and  information  exchanges  identified  in 
the  DoD  Enterprise  Architecture  and  solution 
architectures  based  on  integrated  DODAF  content, 
and  must  satisfy  the  technical  requirements  for 
transition  to  Net-Centric  military  operations  to 
include: 

1)  Solution  architecture  products  compliant  with 
DoD  Enterprise  Architecture  based  on  integrated 
DODAF  content,  including  specified  operationally 
effective  information  exchanges 

2)  Compliant  with  Net-Centric  Data  Strategy  and 
Net-Centric  Services  Strategy,  and  the  principles 
and  rules  identified  in  the  DoD  Information 
Enterprise  Architecture  (DoD  IEA),  excepting 
tactical  and  non-IP  communications 

3)  Compliant  with  GIG  Technical  Guidance  to 
include  IT  Standards  identified  in  the  TV-1  and 
implementation  guidance  of  GIG  Enterprise  Service 
Profiles  (GESPs)  necessary  to  meet  all  operational 
requirements  specified  in  the  DoD  Enterprise 
Architecture  and  solution  architecture  views 

4)  Information  assurance  requirements  including 
availability,  integrity,  authentication, 
confidentiality,  and  non-repudiation,  and  issuance 
of  an  Interim  Authorization  to  Operate  (IATO)  or 
Authorization  to  Operate  by  the  Designated 
Accrediting  Authority  (DAA),  and 

5)  Supportability  requirements  to  include  SAASM, 
Spectrum  and  JTRS  requirements. 


Qhiectivt 


The  capability,  system, 
support  execution  of  all 
information  excharn 
Enterprise  Archit< 
based  on  integrated  DOD/ 
satisfy  the  technical  requ/ 

Net-Centric  military  opc 

1)  Solution  architecture  proc 
DoD  Enterprise  Architecture/ 

DODAF  content,  including 
effective  information  exchahges 

2)  Compliant  with  Net-Centric  Data  Strategy  and 
Net-Centric  Services  Strategy,  and  the  principles 
and  rules  identified  in  the  DoD  Information 
Enterprise  Architecture  (DoD  IEA),  excepting 
tactical  and  non-IP  communications 

3)  Compliant  with  GIG  Technical  Guidance  to 
include  IT  Standards  identified  in  the  TV-1  and 
implementation  guidance  of  GIG  Enterprise  Service 
Profiles  (GESPs)  necessary  to  meet  all  operational 
requirements  specified  in  the  DoD  Enterprise 
Architecture  and  solution  architecture  views 


<sed  d  Ategk  Id 
deified  Operation.  Illy 


4)  Information  assurance  requirements  including 
availability,  integrity,  authentication, 
confidentiality,  and  non-repudiation,  and  issuance 
of  an  Interim  Authorization  to  Operate  (IATO)  or 
Authorization  to  Operate  by  the  Designated 
Accrediting  Authority  (DAA),  and 

5)  Supportability  requirements  to  include  SAASM, 
Spectrum  and  JTRS  requirements. 


niRA  NR-KPP  Requirements 

^  Source  v 

v 


A  Combat  Support  Agency 


O 


[CD 


ODD 


CPD 


ISP 


TISP 


ISP 

Annex 

(Sve  si 

Appsj 


-Q  c 
re  ns 

o 

CL  E 
CL  O 

=  O 
CO 


Note  1 


Note  2 


Note  3 


Note  4 


Note  5 


DOD  Enterprise  Architecture  Products  (IAW  DODAF)  (see  Note  5) 


t-  « 


o 


(O 


<N 


CO 


CO 


(O 


CO 


CD 


to 


£0 


Required  (PM  needs  to  check  with  their  Component  for  any  additional  architectural/regulatory  requirements  for 
CDDs,  CPDs,  ISPs/TISPs.  {e.g.,  HQDA  requires  the  SV-IOc) 


Required  only  when  IT  and  NSS  collects,  processes,  or  uses  any  shared  data  or  when  IT  and  NSS  exposes, 
consumes  or  implements  shared  services, 


The  TV-1  and  TV-2  are  built  using  the  DKSRonline  and  must  be  posted  for  compliance. 


The  AV-1  must  be  uploaded  onto  DARS  and  must  be  registered  in  DARS  for  compliance 


Only  required  for  Milestone  C,  if  applicable  (see  Note  1) 


The  naming  of  the  architecture  views  is  expected  to  change  with  the  release  of  DODAF  v2.Q  (e.g.,  StdV,  SvcV, 
StdV,  DIV).  The  requirements  of  this  matrix  will  not  change.. 
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Mapping  NR-KPP  to 
Operational  Impact 


Enterprise 

Level 

Mission 

Threads 

(JMTs) 


System 

Level 

Operational 

Activities 

(OV-5) 


System 

Level 

Operational 

Exchanges 

(OV-3) 


System 

Level 

Technical 

Exchanges 

(SV-6) 


Shared 
Data  & 
Services 
(EVTS) 


System 

Standards 

(TV-1) 


Joint  Critical 
Technical  Info 
Exchanges (P2P) 


High-Risk 

Standards 


Low-Risk 

Standards 


Operational 

Activities 


K1, 


Joint  Critical 
Operational 
Activities 


Other 

Operational 

Activities 


Joint  Critical 
Operational  Info 
Exchanges 


Other 

Operational  Info 
Exchanges 


Joint  Critical 
Technical  Info 
Exchanges (NC) 


Other  Technical 
Info  Exchanges 


Other  Technical 
Info  Exchanges 


Data  Assets  & 
Net-Centric 
Services 


K1, 


High-Risk 

Standards 


Low-Risk 

Standards 


Universal  Joint 
Task  Metrics 


Universal  Joint 
&  Service 
Task  Metrics 


Secure,  Timely, 
Accurate,  and 
Usable  as 
Defined  in  OV-3 


Secure,  Timely, 
Accurate,  and 
Usable  as 
Defined  in  SV-6 


Visible,  Accessible, 
Understandable, 
Secure, 
Interoperable 


No  Conformance 
Issues  Resulting  in 
Critical  Operational 
Impact 
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Operationally  Effective 
Information  Exchanges 


Net-Ready:  The 
capability,  system,  and/or 
service  must  support  Net- 
Centric  military 
operations.  The 
capability,  system,  and/or 
service  must  be  able  to 
enter  and  be  managed  in 
the  network,  and 
exchange  data  in  a 
secure  manner  to 
enhance  mission 
effectiveness.  The 
capability,  system,  and/or 
service  must 
continuously  provide 
survivable,  interoperable, 
secure,  and  operationally 
effective  information 
exchanges  to  enable  a 
Net-Centric  military 
capability. 


Threshold 


The  capability,  system,  and/or  service  must  fully 
support  execution  of  joint  critical  operational 
activities  and  information  exchanges  identified  in 
the  DoD  Enterprise  Architecture  and  solution 
architectures  based  on  integrated  DODAF  content, 
and  must  satisfy  the  technical  requirements  for 
transition  to  Net-Centric  military  operations  to 
include: 

1)  Solution  architecture  products  compliant  with 
DoD  Enterprise  Architecture  based  on  integrated 
DODAF  content,  including  specified  operationally 
effective  information  exchanges 

2)  Compliant  with  Net-Centric  Data  Strategy  and 
Net-Centric  Services  Strategy,  and  the  principles 
and  rules  identified  in  the  DoD  Information 
Enterprise  Architecture  (DoD  IEA),  excepting 
tactical  and  non-IP  communications 

3)  Compliant  with  GIG  Technical  Guidance  to 
include  IT  Standards  identified  in  the  TV-1  and 
implementation  guidance  of  GIG  Enterprise  Service 
Profiles  (GESPs)  necessary  to  meet  all  operational 
requirements  specified  in  the  DoD  Enterprise 
Architecture  and  solution  architecture  views 

4)  Information  assurance  requirements  including 
availability,  integrity,  authentication, 
confidentiality,  and  non-repudiation,  and  issuance 
of  an  Interim  Authorization  to  Operate  (IATO)  or 
Authorization  to  Operate  by  the  Designated 
Accrediting  Authority  (DAA),  and 

5)  Supportability  requirements  to  include  SAASM, 
Spectrum  and  JTRS  requirements. 


Objective 


The  capability,  system,  and/or  service  must  fully 
support  execution  of  all  operational  activities  and 
information  exchanges  identified  in  the  DoD 
Enterprise  Architecture  and  solution  architectures 
based  on  integrated  DODAF  content,  and  must 
satisfy  the  technical  requirements  for  transition  to 
Net-Centric  military  operations  to  include: 

1)  Solution  architecture  products  compliant  with 
DoD  Enterprise  Architecture  based  on  integrated 
DODAF  content,  including  specified  operationally 
effective  information  exchanges 

2)  Compliant  with  Net-Centric  Data  Strategy  and 
Net-Centric  Services  Strategy,  and  the  principles 
and  rules  identified  in  the  DoD  Information 
Enterprise  Architecture  (DoD  IEA),  excepting 
tactical  and  non-IP  communications 

3)  Compliant  with  GIG  Technical  Guidance  to 
include  IT  Standards  identified  in  the  TV-1  and 
implementation  guidance  of  GIG  Enterprise  Service 
Profiles  (GESPs)  necessary  to  meet  all  operational 
requirements  specified  in  the  DoD  Enterprise 
Architecture  and  solution  architecture  views 

4)  Information  assurance  requirements  including 
availability,  integrity,  authentication, 
confidentiality,  and  non-repudiation,  and  issuance 
of  an  Interim  Authorization  to  Operate  (IATO)  or 
Authorization  to  Operate  by  the  Designated 
Accrediting  Authority  (DAA),  and 

5)  Supportability  requirements  to  include  SAASM, 
Spectrum  and  JTRS  requirements. 
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Operationally  Effective 
Information  Exchanges 

•  Requirements  Analysis 

-  What  missions  and  activities  does  the  system  support? 

-  Joint  Mission  Threads 

-  OV-6c 

-  OV-5 

-  What  information  exchanges  are  necessary  to  execute  those 
missions  and  activities? 

-  OV-3 

-  SV-6 

•  Test  Planning  and  Execution 

-  Must  be  on  production  representative  system  in  an  operationally 
realistic  environment 


A  Combat  Support  Agency 


Did  the  system  meet  all  joint  critical  information  exchange  requirements? 


Net-Ready:  The 
capability,  system,  and/or 
service  must  support  Net- 
Centric  military 
operations.  The 
capability,  system,  and/or 
service  must  be  able  to 
enter  and  be  managed  in 
the  network,  and 
exchange  data  in  a 
secure  manner  to 
enhance  mission 
effectiveness.  The 
capability,  system,  and/or 
service  must 
continuously  provide 
survivable,  interoperable, 
secure,  and  operationally 
effective  information 
exchanges  to  enable  a 
Net-Centric  military 
capability. 


Threshold 


The  capability,  system,  and/or  service  must  fully 
support  execution  of  joint  critical  operational 
activities  and  information  exchanges  identified  in 
the  DoD  Enterprise  Architecture  and  solution 
architectures  based  on  integrated  DODAF  content, 
and  must  satisfy  the  technical  requirements  for 
transition  to  Net-Centric  military  operations  to 
include: 

1)  Solution  architecture  products  compliant  with 
DoD  Enterprise  Architecture  based  on  integrated 

DODAF  content,  including  specified  operationally 
effective  information  exchanges 

2)  Compliant  with  Net-Centric  Data  Strategy  and 
Net-Centric  Services  Strategy,  and  the  principles 
and  rules  identified  in  the  DoD  Information 
Enterprise  Architecture  (DoD  IEA),  excepting 
tactical  and  non-IP  communications 

3)  Compliant  with  GIG  Technical  Guidance  to 
include  IT  Standards  identified  in  the  TV-1  and 
implementation  guidance  of  GIG  Enterprise  Service 
Profiles  (GESPs)  necessary  to  meet  all  operational 
requirements  specified  in  the  DoD  Enterprise 
Architecture  and  solution  architecture  views 

4)  Information  assurance  requirements  including 
availability,  integrity,  authentication, 
confidentiality,  and  non-repudiation,  and  issuance 
of  an  Interim  Authorization  to  Operate  (IATO)  or 
Authorization  to  Operate  by  the  Designated 
Accrediting  Authority  (DAA),  and 

5)  Supportability  requirements  to  include  SAASM, 
Spectrum  and  JTRS  requirements. 


Objective 


The  capability,  system,  and/or  service  must  fully 
support  execution  of  all  operational  activities  and 
information  exchanges  identified  in  the  DoD 
Enterprise  Architecture  and  solution  architectures 
based  on  integrated  DODAF  content,  and  must 
satisfy  the  technical  requirements  for  transition  to 
Net-Centric  military  operations  to  include: 

1)  Solution  architecture  products  compliant  with 
DoD  Enterprise  Architecture  based  on  integrated 
DODAF  content,  including  specified  operationally 
effective  information  exchanges 

2)  Compliant  with  Net-Centric  Data  Strategy  and 
Net-Centric  Services  Strategy,  and  the  principles 
and  rules  identified  in  the  DoD  Information 
Enterprise  Architecture  (DoD  IEA),  excepting 
tactical  and  non-IP  communications 

3)  Compliant  with  GIG  Technical  Guidance  to 
include  IT  Standards  identified  in  the  TV-1  and 
implementation  guidance  of  GIG  Enterprise  Service 
Profiles  (GESPs)  necessary  to  meet  all  operational 
requirements  specified  in  the  DoD  Enterprise 
Architecture  and  solution  architecture  views 

4)  Information  assurance  requirements  including 
availability,  integrity,  authentication, 
confidentiality,  and  non-repudiation,  and  issuance 
of  an  Interim  Authorization  to  Operate  (IATO)  or 
Authorization  to  Operate  by  the  Designated 
Accrediting  Authority  (DAA),  and 

5)  Supportability  requirements  to  include  SAASM, 
Spectrum  and  JTRS  requirements. 
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Data  Strategy  Compliance 

Visible 
Accessible 
Data  Management 
Understandable 
Trusted 
Interoperable 

Responsive  to  User’s  Needs 


Services  Strategy  Compliance 

Provide  Services 
Use  Services 

Govern  the  Infrastructure  and  Services 
Monitor  and  Manage  Services  via  GIG  NetOPS 


DoD  Information  Enterprise  Architecture  Compliance 

Data  and  Services  Deployment 
Secured  Availability 
Shared  Infrastructure  Environment 
Computing  Infrastructure  Readiness 
NetOPS  Agility 
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Data  Strategy  Compliance 

visioie 

Accessible 

Data  Management 
Understandable 

Trusted 

Interoperable 

Responsive  to  User’s  Needs 

“Discovery  Metadata  shall 
conform  to  DDMS” 


“Post  descriptions  of  access 
mechanisms” 


“Associate  data  pedigree  metadata” 
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Net-Centric  Data  Requirement 

Data  is  Visible 

Post  discovery  metadata  in  an  Enterprise  Catalog:  Department  of  Defense  (DoD)  Discovery 
Metadata  Specification  (DDMS)-  conformant  discovery  metadata  is  posted  in  the 
Net-Centric  Enterprise  Services  (NCES)  Enterprise  Catalog  or  other  compatible/federated 
enterprise  catalog  that  is  visible  to  the  Enterprise. 

Use  appropriate  keywords  for  discovery:  Discovery  keywords  should  reflect  common  user 
terms,  be  appropriate  for  mission  area  or  data  type,  be  understandable,  and  conform  with 
MDR  requirements  that  map  back  to  COI  identified  mission  data. _ 

Data  is  Accessible 

Post  data  to  shared  space:  Data  asset  is  available  in  a  shared  space,  i.e. ,  a  space  that  is 
accessible  to  multiple  end  users. 

Provide  access  policy:  If  data  is  not  accessible  to  all  users,  a  written  policy  on  how  to  gain 
access  is  available  and  accurate. 

Provide  serving  (access)  mechanism:  Shared  space  provides  serving  (access)  mechanisms 
for  the  data.  I.e.,  a  service  provides  users  with  access  to  the  data. 

Publish  active  link  to  data  asset:  The  Enterprise  Catalog  DoD  Discovery  Metadata 
Specification  (DDMS)  entry  contains  an  active  link  (e.g.,  Uniform  Resource  Identifier  (URI)) 
to  the  data  asset. _ 

Data  is  Understandable 

Publish  semantic  and  structural  metadata 

-  Semantic  and  structural  metadata  are  published  in  the  Enterprise  Catalog. 

Register  data  artifacts  in  DoD  MDR 

-  XML  schema  definitions  (XSD),  extensible  Markup  Language  (XML)  instances,  data 

models  (such  as  entity  relationship  diagrams)  and  other  appropriate  artifacts  are  registered 
in  the  DoD  Metadata  Registry  (MDR). _ 

Data  is  Interoperable 

Base  vocabularies  on  Universal  Core  (UCore) 

-  Semantic  vocabularies  reuse  elements  of  the  Universal  Core  (Ucore)  standard. 

Comply  with  COI  data-sharinq  agreements 

-  Semantic  and  structural  metadata  conform  to  interoperability  agreements  promoted 
through  communities,  e.g.,  Community  of  Interest  (COI). 

Conform  to  DDMS 

-  All  metadata,  including  record-level  database  tagging  and  in-line  document 

tagging,  complies  with  DDMS. _ 

Data  is  Trusted 

Provide  information  assurance  and  security  metadata 

-  All  metadata,  including  record-level  database  tagging  and  in-line  document 

tagging,  includes  data  pedigree  and  security  metadata,  as  well  as  an  authoritative  source  for 
the  data  (when  appropriate). _ 


Net-Centric  Services  Requirement 

Services  are  Visible 

Publish  a  description  of  the  service  or  access  mechanism 

-  Descriptions  (metadata)  for  the  service  or  access  mechanism  are  published  in  an 
enterprise  service  registry,  e.g.,  the  NCES  Service  Registry. 

Comply  with  enterprise-specified  minimum  service  discovery  requirements 

-  The  data  access  mechanism  complies  with  enterprise-specified  minimum  service 
discovery  requirements,  e.g.,  a  Universal  Description,  Discovery  and  Integration  (UDDI) 
description  to  enable  federated  discovery. 

Services  are  Accessible 

Provide  an  active  link  to  the  service  in  the  enterprise  catalog 

-  Active  link  (e.g.,  Uniform  Resource  Identifier  (URI))  to  the  specified  service  is  included 
in  the  enterprise  catalog  metadata  entry  (i.e.,  metacard)  for  the  specified  service. 

Provide  an  active  link  to  the  service  in  the  NCES  Service  Reqistrv 

-  URIs  as  the  operational  end  points  for  services  shall  be  registered  in  the  NCES  Service 
Registry  by  referencing  the  WSDL  (that  is  in  the  MDR). 

Services  are  Understandable 

Publish  a  description  of  the  service  or  access  mechanism  to  the  NCES  Service  Reqistrv 

-  Metadata  for  the  service  or  access  mechanism  are  published  in  the  NCES  Service 
Registry. 

Publish  service  artifacts  to  DoD  MDR 

-  Web  Service  Description  Language  (WSDL)  documents,  and  other  appropriate  artifacts 
are  registered  in  the  DoD  Metadata  Registry  (MDR). 

Provide  service  specification  or  Service  Level  Agreement  (SLA) 

-  A  service  specification  or  Service  Level  Agreement  (SLA)  exists  for  services  and  data 
access  mechanisms. 

Services  are  Trusted 

Operate  services  in  accordance  with  SLA 
-  The  service  meets  the  performance  standards  in  the  SLA 

Include  security  mechanisms  or  restrictions  in  the  service  specification 

-  The  service  specification  describes  security  mechanisms  or  restrictions  that  apply  to 
the  service 

Enable  continuity  of  operations  and  disaster  recovery  for  services 

-  The  service  has  a  defined  and  functional  Continuity  of  Operations  Plan 

Provide  NetOps  Data  (NetOps  Aqilitv) 

-  Services  and  data  access  mechanisms  provide  operational  states,  performance, 
availability,  and  security  data/information  to  NetOps  management  services,  e.g., 

Enterprise  Management,  Content  Management,  and  Network  Defense  services 

Use  of  Core  Enterprise  Services  (CES) 

-  Core  Enterprise  Services  (CES)  are  used  in  accordance  with  DoD  CIO  mandates 
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*  Requirements  Analysis 

-  Do  the  net-centric  requirements  apply? 

-  What  enterprise-level  shared  data  and  service  assets  are 
documented  in  the  Exposure  Verification  Tracking  Sheets? 

-  What  data  and  service  assets  support  a  joint  critical 
operational  activity? 

*  Test  Planning  and  Execution 

-  Static  analysis  (e.g.,  registration  of  assets) 

-  Conformance/compliance  testing  (e.g.,  schema 
conformance) 

-  Mission  effectiveness  (e.g.,  visibility,  accessibility) 


Did  the  system  meet  all  joint  critical  net-centric  requirements? 
(Visible,  Accessible,  Understandable,  Trusted,  Interoperable) 


A  Combat  Support  Agency 


GIG  Technical  Guidance 


Net-Ready:  The 
capability,  system,  and/or 
service  must  support  Net- 
Centric  military 
operations.  The 
capability,  system,  and/or 
service  must  be  able  to 
enter  and  be  managed  in 
the  network,  and 
exchange  data  in  a 
secure  manner  to 
enhance  mission 
effectiveness.  The 
capability,  system,  and/or 
service  must 
continuously  provide 
survivable,  interoperable, 
secure,  and  operationally 
effective  information 
exchanges  to  enable  a 
Net-Centric  military 
capability. 


Threshold 


The  capability,  system,  and/or  service  must  fully 
support  execution  of  joint  critical  operational 
activities  and  information  exchanges  identified  in 
the  DoD  Enterprise  Architecture  and  solution 
architectures  based  on  integrated  DODAF  content, 
and  must  satisfy  the  technical  requirements  for 
transition  to  Net-Centric  military  operations  to 
include: 

1)  Solution  architecture  products  compliant  with 
DoD  Enterprise  Architecture  based  on  integrated 
DODAF  content,  including  specified  operationally 
effective  information  exchanges 

2)  Compliant  with  Net-Centric  Data  Strategy  and 
Net-Centric  Services  Strategy,  and  the  principles 
and  rules  identified  in  the  DoD  Information 
Enterprise  Architecture  (DoD  IEA),  excepting 
tactical  and  non-IP  communications 

3)  Compliant  with  GIG  Technical  Guidance  to 
include  IT  Standards  identified  in  the  TV-1  and 
implementation  guidance  of  GIG  Enterprise  Service 
Profiles  (GESPs)  necessary  to  meet  all  operational 
requirements  specified  in  the  DoD  Enterprise 
Architecture  and  solution  architecture  views 

4)  Information  assurance  requirements  including 
availability,  integrity,  authentication, 
confidentiality,  and  non-repudiation,  and  issuance 
of  an  Interim  Authorization  to  Operate  (IATO)  or 
Authorization  to  Operate  by  the  Designated 
Accrediting  Authority  (DAA),  and 

5)  Supportability  requirements  to  include  SAASM, 
Spectrum  and  JTRS  requirements. 


Objective 


The  capability,  system,  and/or  service  must  fully 
support  execution  of  all  operational  activities  and 
information  exchanges  identified  in  the  DoD 
Enterprise  Architecture  and  solution  architectures 
based  on  integrated  DODAF  content,  and  must 
satisfy  the  technical  requirements  for  transition  to 
Net-Centric  military  operations  to  include: 

1)  Solution  architecture  products  compliant  with 
DoD  Enterprise  Architecture  based  on  integrated 
DODAF  content,  including  specified  operationally 
effective  information  exchanges 

2)  Compliant  with  Net-Centric  Data  Strategy  and 
Net-Centric  Services  Strategy,  and  the  principles 
and  rules  identified  in  the  DoD  Information 
Enterprise  Architecture  (DoD  IEA),  excepting 
tactical  and  non-IP  communications 

3)  Compliant  with  GIG  Technical  Guidance  to 
include  IT  Standards  identified  in  the  TV-1  and 
implementation  guidance  of  GIG  Enterprise  Service 
Profiles  (GESPs)  necessary  to  meet  all  operational 
requirements  specified  in  the  DoD  Enterprise 
Architecture  and  solution  architecture  views 

4)  Information  assurance  requirements  including 
availability,  integrity,  authentication, 
confidentiality,  and  non-repudiation,  and  issuance 
of  an  Interim  Authorization  to  Operate  (IATO)  or 
Authorization  to  Operate  by  the  Designated 
Accrediting  Authority  (DAA),  and 

5)  Supportability  requirements  to  include  SAASM, 
Spectrum  and  JTRS  requirements. 
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DISA  gig 

A  Combat  Support  Agency 


Technical  Guidance 

Standards  Conformance 


•  Requirements  Analysis 

-  Risk  analysis  on  standards 
identified  in  system  TV-1 


•  Test  Planning  and  Execution 

-  Leverage  commercial  and 
government  test  results,  as 
appropriate 

-  Execute  standards 
conformance  testing,  as 
appropriate 


Manage  List  of  Standards  or  TV- Is 


|  Add  New  List/TV-1 


c 


Save  | 


View  All  Lists/TV-ls 


I  JITC  Risk  Report  for  this  List/TV- 1 


Or  Select  a  TV-1:  CFMS  TV-1  (Group:  CFMS  ;  CSS:  System) 
Description: 


In  addition  to  standards  shown  for  this  list  of  standards  or  TV-1,  you  can  also  get  standards  from  the  following. 


From  Another  List/TV-1 : 
From  an  Interest  Group: 


v]  |  Get  It!  | 


[  Get  Emerging  and  Mandated  DISR  Standards  in  this  Database!  | 


,V.|  I  Get  It!  | 


Get  All  Standards  in  this  Database! 


Only  CHECKED  items  will  be  saved  on  your  list  of  standards  or  TV-1.  Unchecked  items  will  be  removed  from  your  list  but  can  be  reselected. 


►  El  ANSI  T 1 . 1 02- 1 993  (R2005)  Digital  Hierarchy  -  Electrical  Interfaces,  December  1 993 


0  CIM  HTTP 


Specification  for  CIM  Operations  over  HTTP  Version  1.0,  Distributed  Management  Task  Force,  Inc.,  11  August  1999 


0  CIM  XML 


Specifi  i  i  i  i  i  r  Manag  T  -  l  F  r  ^  Ir  £0 


M  |CI5SISM^XML  Common  Information  Sharing  Standard  for  Information  Security  Marking:  XML  Implementation,  Implementation  Guide,  Release  2.0.3,  15  February  20\ 

1^1  |Core^Architecture^ataModeKC|^|Core  Architecture  Data  Model  (CADM)  Baseline  1 .03,  9  May  2006  | 


F7|  Core  Architecture  Data  Model  (C.  Core  Architecture  Data  Model  (CADM)  Baseline  1 .03,  9  May  2006 


0  FIPS  Pub  10-4:2002 


]  Countries,  Dependencies,  Areas  of  Special  sovereignty,  and  their  Principle  Administrative  Divisions,  April  1995  as  modified  by  various  Change  Notices| 


STANDARDS 


Version:  01 .01 .01 , 03/06/2009  20:20  MST 
Currently  linked  to  backend  database:  D:\BACKUP\DBs\U_SGSDB_be_Jest.mdb 

Welcome  Sivey  Anderson 

ADD/EDIT  QUERIES 


Test  Methodologies 


Products  Test  Tools 


Interest  Groups 


Testing  Organizations 


|  Open  Links  to  Sources  ~ 


NCGIS  Stoplight  Form 


List  of  Standards  or  TV-I 


Standards  Table  of  Contents 


Standards  with  JITC  Risk  Rationale  | 


Database  Information:  Sivey  Anderson,  520-538-0786,  Sivey. Anderson. ctr@>disa.m 


Open  the  User's  Manual 


Did  the  system  have  any  conformance-issues  that  could 
result  in  a  critical  operational  impact? 
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A  Combat  Support  Agency 


Information  Assurance 


Net-Ready:  The 
capability,  system,  and/or 
service  must  support  Net- 
Centric  military 
operations.  The 
capability,  system,  and/or 
service  must  be  able  to 
enter  and  be  managed  in 
the  network,  and 
exchange  data  in  a 
secure  manner  to 
enhance  mission 
effectiveness.  The 
capability,  system,  and/or 
service  must 
continuously  provide 
survivable,  interoperable, 
secure,  and  operationally 
effective  information 
exchanges  to  enable  a 
Net-Centric  military 
capability. 


Threshold 


The  capability,  system,  and/or  service  must  fully 
support  execution  of  joint  critical  operational 
activities  and  information  exchanges  identified  in 
the  DoD  Enterprise  Architecture  and  solution 
architectures  based  on  integrated  DODAF  content, 
and  must  satisfy  the  technical  requirements  for 
transition  to  Net-Centric  military  operations  to 
include: 

1)  Solution  architecture  products  compliant  with 
DoD  Enterprise  Architecture  based  on  integrated 
DODAF  content,  including  specified  operationally 
effective  information  exchanges 

2)  Compliant  with  Net-Centric  Data  Strategy  and 
Net-Centric  Services  Strategy,  and  the  principles 
and  rules  identified  in  the  DoD  Information 
Enterprise  Architecture  (DoD  IEA),  excepting 
tactical  and  non-IP  communications 

3)  Compliant  with  GIG  Technical  Guidance  to 
include  IT  Standards  identified  in  the  TV-1  and 
implementation  guidance  of  GIG  Enterprise  Service 
Profiles  (GESPs)  necessary  to  meet  all  operational 
requirements  specified  in  the  DoD  Enterprise 
Architecture  and  solution  architecture  views 

4)  Information  assurance  requirements  including 
availability,  integrity,  authentication, 
confidentiality,  and  non-repudiation,  and  issuance 
of  an  Interim  Authorization  to  Operate  (IATO)  or 
Authorization  to  Operate  by  the  Designated 
Accrediting  Authority  (DAA),  and 

5)  Supportability  requirements  to  include  SAASM, 
Spectrum  and  JTRS  requirements. 


Objective 


The  capability,  system,  and/or  service  must  fully 
support  execution  of  all  operational  activities  and 
information  exchanges  identified  in  the  DoD 
Enterprise  Architecture  and  solution  architectures 
based  on  integrated  DODAF  content,  and  must 
satisfy  the  technical  requirements  for  transition  to 
Net-Centric  military  operations  to  include: 

1)  Solution  architecture  products  compliant  with 
DoD  Enterprise  Architecture  based  on  integrated 
DODAF  content,  including  specified  operationally 
effective  information  exchanges 

2)  Compliant  with  Net-Centric  Data  Strategy  and 
Net-Centric  Services  Strategy,  and  the  principles 
and  rules  identified  in  the  DoD  Information 
Enterprise  Architecture  (DoD  IEA),  excepting 
tactical  and  non-IP  communications 

3)  Compliant  with  GIG  Technical  Guidance  to 
include  IT  Standards  identified  in  the  TV-1  and 
implementation  guidance  of  GIG  Enterprise  Service 
Profiles  (GESPs)  necessary  to  meet  all  operational 
requirements  specified  in  the  DoD  Enterprise 
Architecture  and  solution  architecture  views 

4)  Information  assurance  requirements  including 
availability,  integrity,  authentication, 
confidentiality,  and  non-repudiation,  and  issuance 
of  an  Interim  Authorization  to  Operate  (IATO)  or 
Authorization  to  Operate  by  the  Designated 
Accrediting  Authority  (DAA),  and 

5)  Supportability  requirements  to  include  SAASM, 
Spectrum  and  JTRS  requirements. 
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OKA 

A  Combat  Support  Agency 


Information  Assurance 


*  Requirements  Analysis 

-  What  Certification  and  Accreditation  (C&A)  process  (DIACAP, 
NISCAP,  ICD  503)  does  the  system  fall  under? 

*  Test  Planning  and  Execution 

-  Ensure  the  system  is  operating  in  the  approved  IA  configuration 
during  interoperability/operational  testing 

-  Verify  IATO/ATO 

-  Execute  required  additional  IA  testing 


Has  the  system  received  an  Interim  Authority  to 
Operate  (IATO)/Authority  to  Operate  (ATO)? 


A  Combat  Support  Agency 


Suppo  liability 


Net-Ready:  The 
capability,  system,  and/or 
service  must  support  Net- 
Centric  military 
operations.  The 
capability,  system,  and/or 
service  must  be  able  to 
enter  and  be  managed  in 
the  network,  and 
exchange  data  in  a 
secure  manner  to 
enhance  mission 
effectiveness.  The 
capability,  system,  and/or 
service  must 
continuously  provide 
survivable,  interoperable, 
secure,  and  operationally 
effective  information 
exchanges  to  enable  a 
Net-Centric  military 
capability. 


Threshold 


The  capability,  system,  and/or  service  must  fully 
support  execution  of  joint  critical  operational 
activities  and  information  exchanges  identified  in 
the  DoD  Enterprise  Architecture  and  solution 
architectures  based  on  integrated  DODAF  content, 
and  must  satisfy  the  technical  requirements  for 
transition  to  Net-Centric  military  operations  to 
include: 

1)  Solution  architecture  products  compliant  with 
DoD  Enterprise  Architecture  based  on  integrated 
DODAF  content,  including  specified  operationally 
effective  information  exchanges 

2)  Compliant  with  Net-Centric  Data  Strategy  and 
Net-Centric  Services  Strategy,  and  the  principles 
and  rules  identified  in  the  DoD  Information 
Enterprise  Architecture  (DoD  IEA),  excepting 
tactical  and  non-IP  communications 

3)  Compliant  with  GIG  Technical  Guidance  to 
include  IT  Standards  identified  in  the  TV-1  and 
implementation  guidance  of  GIG  Enterprise  Service 
Profiles  (GESPs)  necessary  to  meet  all  operational 
requirements  specified  in  the  DoD  Enterprise 
Architecture  and  solution  architecture  views 

4)  Information  assurance  requirements  including 
availability,  integrity,  authentication, 
confidentiality,  and  non-repudiation,  and  issuance 
of  an  Interim  Authorization  to  Operate  (IATO)  or 
Authorization  to  Operate  by  the  Designated 
Accrediting  Authority  (DAA),  and 

5)  Supportability  requirements  to  include  SAASM, 
Spectrum  and  JTRS  requirements. 


Objective 


The  capability,  system,  and/or  service  must  fully 
support  execution  of  all  operational  activities  and 
information  exchanges  identified  in  the  DoD 
Enterprise  Architecture  and  solution  architectures 
based  on  integrated  DODAF  content,  and  must 
satisfy  the  technical  requirements  for  transition  to 
Net-Centric  military  operations  to  include: 

1)  Solution  architecture  products  compliant  with 
DoD  Enterprise  Architecture  based  on  integrated 
DODAF  content,  including  specified  operationally 
effective  information  exchanges 

2)  Compliant  with  Net-Centric  Data  Strategy  and 
Net-Centric  Services  Strategy,  and  the  principles 
and  rules  identified  in  the  DoD  Information 
Enterprise  Architecture  (DoD  IEA),  excepting 
tactical  and  non-IP  communications 

3)  Compliant  with  GIG  Technical  Guidance  to 
include  IT  Standards  identified  in  the  TV-1  and 
implementation  guidance  of  GIG  Enterprise  Service 
Profiles  (GESPs)  necessary  to  meet  all  operational 
requirements  specified  in  the  DoD  Enterprise 
Architecture  and  solution  architecture  views 

4)  Information  assurance  requirements  including 
availability,  integrity,  authentication, 
confidentiality,  and  non-repudiation,  and  issuance 
of  an  Interim  Authorization  to  Operate  (IATO)  or 
Authorization  to  Operate  by  the  Designated 
Accrediting  Authority  (DAA),  and 

5)  Supportability  requirements  to  include  SAASM, 
Spectrum  and  JTRS  requirements. 
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Suppo  liability 


•  Threshold  =  Objective 

-  Spectrum  Supportability 

o  Verify  the  system  has  an  approved  (Stage  4)  DD  Form  1494 
(for  any  spectrum  dependent  system)  (DoDI  5000.02) 

o  Verify  completion  of  applicable  requirements  of  DODD 
3222.2,  “DOD  Electromagnetic  Environmental  Effects  (E3)” 

-  Selective  Availability  Anti-Spoofing  Module  (SAASM) 

o  Verify  any  GPS  receivers  procured  are  SAASM  compliant  or 
that  a  waiver  has  been  obtained  from  ASD(NII) 

-  Joint  Tactical  Radio  System  (JTRS) 

o  Verify  a  JTRS  solution  or  waiver  from  ASD(NII)  for  any  radio 
solution  operating  within  the  2MHz  to  2  GHz  range* 


*Reference:  (ASD(NII)/DOD  CIO  memorandum ,  23  May  2005,  “Temporary  Suspension  of  the 
Joint  Tactical  Radio  Systems  (JTRS)  Waiver  Process”  and  ASD(NII)/DOD  CIO  memorandum, 
12  January  2007  “Reinstatement  of  the  Joint  Tactical  Radio,  (JTRS)  Waiver  Process  for 
Handheld  Radio  Procurements”) 
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Interoperability 
Certification  Products 


Certification 

Description 

System  can  be 
fielded  (Y/N)? 

Standards  Conformance 
Certification 

System  is  certified  for  conformance  to  a 
standard/standards  profile 

No 

Joint  Interoperability  Test 
Certification 

Full  system  certification.  System  meets  at  least  all 
critical  interoperability  requirements 

Yes 

Limited  Joint  Interoperability 
Test  Certification 

System  meets  subset  of  critical  interoperability 
requirements 

Yes,  with  ICTO 

Interim  Joint  Interoperability 
Test  Certification 

Capability  module  has  adequately  demonstrated 
interoperability  for  at  least  all  critical  threshold 
requirements  identified  for  the  increment 

Yes 

Special  Interoperability  Test 
Certification 

Certification  is  based  on  other  J-6  approved 
requirements  other  than  the  NR-KPP,  e.g.,  use  of  UCR 
for  voice  switches 

Yes 

Non-Certification 

Critical  operational  impacts  expected 

Provides  a  warning  to  the  warfighter 

No 

Interoperability  Assessment 

PM  would  like  to  determine  interoperability  status. 
System  may  lack  J-6  certified  requirements 

No 

A  Combat  Support  Agency 


Contact  Information  & 
Resources 


Hotline 

-  24/7  C4I  Technical  Support 

-  1-800-538-JITC  (5482) 

-  hotline@disa.mil 

-  http://jitc.fhu.disa.mil/support.html 
Joint  Interoperability  Tool  (JIT) 

-  http://jit.fhu.disa.mil 

-  Lessons  Learned  reports 

-  NATO  Interface  Guide 
System  Tracking  Program  (STP) 

-  https://stp.fhu.disa.mil 

-  Test  events 

-  Test  plans  and  reports 

-  Certification  results 
NR-KPP  Helpdesk 

-  NR-KPP  Helpdesk@disa.mil 
NR-KPP  Testing  Guidebook 

-  https  ://www.  us  .army,  m  i  l/su  ite/doc/23429848 
CJCSI  6212  Resource  Page 

-  https://www.intelink.gOv/wiki/Portal:CJCSI_6212 
Resource_Page 
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Questions? 


Danielle  Mackenzie  Koester 
Chief,  Engineering  &  Policy  Branch 
Joint  Interoperability  Test  Command 
March  15,  2011 
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A  Proposal  for  Robotic  Entity  Safety  Release 

Definitions 


VG-51 3  JVM  in  FL,  T  &  E  Conference  March  201 1  (March  3,  201 1  dlf) 


OptiMetrics,  Inc.  Proprietary 


A  Proposal  for  Robotic  Entity  Safety  Release 


Objective 


•  To  reduce  the  high  risk  nature  for  the  OSD  T&E  safety  releases 
and  confirmations  involving  collaborative  and  autonomous  robotic 
missions  for  the  Armed  Forces 

•  Effective  Risk  Mitigation  Requires  Established: 


•  Measures  of  Performance 

•  Relevant  COICS 

•  Relevant  KPPs 

•  Relevant  TPMs 

•  Ability  to  Reliably  Replicate  the 
Intended  Environment  for  use 
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A  Proposal  for  Robotic  Entity  Safety  Release 


Scope 

There  can  be  a  process  implemented  where  the  affected  Project 
Management  offices,  the  Warfighter,  and  T&E  organizations  can 
utilize  advanced  simulation,  component  level  testing,  and  iterative 
limited  user  testing  to  achieve  the  goal  of  a  full  safety  confirmation 
for  human  and  robotic  collaborative  operations. 


Automate  as  much  testing  as  possible  to  support  T&E  and 

PM  and  Warfighter  requirements 


V 
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A  Proposal  for  Robotic  Entity  Safety  Release 


V 


Proposal 


ESTABLISH 

1)  SOFTWARE 

2)  COMPONENT 

3)  SYSTEM 
PERFORMANCES 


TESTING  FOCUSED 
ON 

RELIABILITY 


QUANTIFY  RISKS 


The  process  not  only  establishes  system  performance,  but 
supports  system  confidence  and  quantifies  system  reliability 
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■  <  r  A  Proposal  for  Robotic  Entity  Safety  Release 


New  Categories  of  Testers/Evaluators 


A  Proposal  for  Robotic  Entity  Safety  Ralaasa 

Challenges 

•  Environment  Accreditation 

•  Exponential  Permutations  in  Software  Code 

•  Potential  for  Concomitant  Affects 

•  Potential  for  Critical  System  Bugs 
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A  Proposal  for  Robotic  Entity  Safety  Release 


Software  Cybernetics 


•  T&E  Scope 

•  Multi-dimensional  Nature 

•  Potentially  non-linear  T&E 

•  Simultaneous  and  Conditional  Channels  of 
Information  (increased  I/O) 

•  Level  of  Cognition 

•  Open  Questions 
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A  Proposal  for  Robotic  Entity  Safety  Ralaasa 


Developmental  Testing 

Successful  Safety  Testing  for  Release  and  Confirmation 
Leads  to  Acceptable  Risk  for  Developmental  Test 

•  Establish  Performance  Envelopes 

•  Assess  Degraded  States/Error  Conditions 

•  Extrapolate  Operational  Profiles 

•  Test  for  Reliability  as  a  Function  of  Capability 

•  Establish  Risk  of  Action  (Correct  and  Incorrect) 

•  Identify  Failures  Impacting  Reliability  of 
Operation 
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A  Proposal  for  Robotic  Entity  Safety  Release 


Something  New 

1)  Assess  Robotic  Behavioral  defects  against  their  impacts  on 
Performance  baseline  and  successively  more  difficult  test 
cases;  Defects  and  anomalies  are  quantified  to  assess  risk 
of  failure  or  range  of  potential  actions  and  their  risks  to 
mission  reliability/success 

2)  The  process  begins  with  software  and  is  continued  for  all 
system  components  and  systems 

3)  The  evaluation  of  the  System  of  Systems  is  accomplished 
through  repetition  of  mission  environments 

4)  Outcomes  establish  mission  norms  and  protocols  for 
operation 


Robotic  behaviors  will  be  synonymous  with  mechanical 

system  function  in  the  future 
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A  Potential  Schama  for  Robotic  Development 

and  Evaluation 


Mission 

Area 
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A  Proposal  for  Robotic  Entity  Safety  Release 
Operational  Test/Usage  Requirements 

•  Getting  the  Right  Technology  to  the  Warfighter 

•  Focused  Operational  Testing 
•Allowing  Conditional  Autonomy 

•  Validation  of  Degraded  States  of  Operation 

•  Trust  and  Confidence  of  Operation  and  Performance 

•  Error  Tolerant  Systems 

•  Ability  to  Adapt  to  Social  Cues 

•  Ability  to  Operate  in  a  Variety  of  Dynamic  Environments 


“Soldiers  must  be  able  to  control  autonomous  systems  to  suite 
conditions  as  they  change  over  time.”  (LTG  Vane  U.S.  Army) 
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■  ^  P  A  Proposal  for  Robotic  Entity  Safety  Release 


New  Tools 


•  Leverage  of  OGA  Technology  Rodeo  and  Challenge  Events 

•  Adaptive  Software  Testing 

•  Use  of  Genetic  Algorithms 

•  Enhanced  Simulation  Environments 

•  Development  of  Reality  Arenas 
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A  Proposal  for  Robotic  Entity  Safety  Release 


Outcome 


The  OSD  T&E  community  will  have  supported  the  development 
and  institutionalization  of  a  repeatable  process  and  robust  tools 
that  can  be  re-used  across  many  robotic  platforms  and 
potentially  provided  to  robot  vendors,  and  usable  by  the  PMO 
for  simulation  based  acquisition 
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A  Proposal  for  Robotic  Entity  Safety  Ralaasa 


Additional  Considerations 


Interactive  Training  (Embedded)  in  Robotic  Systems 
Robotic  Puckstering 
Co-existence  (social/work  networking) 

Far-reach  Maintenance  Operations 
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Defense  Information  Systems  Agency 
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DIS^ 

Defense  Information  Systems  Agency 
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Decoupled  Test, 
Evaluation,  and 
Certification  of  a 
System  of  Systems 

(SoS) 
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OKA 

A  Combat  Support  Agency 


The  information  provided  in  this  briefing  is  for 
general  information  purposes  only.  It  does  not 
constitute  a  commitment  on  behalf  of  the  United 
States  Government  to  provide  any  of  the  capabilities, 
systems  or  equipment  presented  and  in  no  way 
obligates  the  United  States  Government  to  enter  into 
any  future  agreements  with  regard  to  the  same.  The 
information  presented  may  not  be  disseminated 
without  the  express  consent  of  the  United  States 
Government. 
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Overview 


A  Combat  Support  Agency 


*  This  presentation  outlines  the  use  of  agile  testing 
concepts,  decoupled  testing,  to  reduce  risk  and 
accelerate  a  Joint  Interoperability  assessment/ 
characterization  of  a  highly  complex  System  of 
Systems  (SoS)  that  does  not  have  an  overarching 
Joint  Staff  (JS)-approved  set  of  requirements 

*  Our  example  is  the  Ballistic  Missile  Defense  System 
(BMDS),  which  has  no  JS-approved  requirements, 
and  its  elements  (THAAD,  Patriot,  JTAGS,  AEGIS, 
etc.),  which  do  have  JS-approved  requirements 
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Background 


A  Combat  Support  Agency 


•  Secretary  Rumsfeld  memo  dated  2  January  2002: 

-  Directed  that  the  BMDS  will  not  be  subject  to 
the  traditional  requirements  generation 
process 

-  Directed  a  capability-based  requirements 
process  for  MD 

-  Directed  the  Services  to  procure  the  BMDS 
elements 

-  Directed  that  Service  BMDS  elements  will 
enter  the  formal  DoD  acquisition  cycle  at 
MS  C 

-  Directed  use  of  rapid  decision  making  cycles 
for  MD 
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Background  (cont.) 


Individual  elements  of  the  BMDS  have  gone 
through  the  Joint  Capabilities  Integration 
Development  System  (JCIDS)  process  and 
developed  applicable  architecture  documents, 
etc. 

BMDS  has  no  Joint  Staff  (JS)-approved 
requirements  and  no  intention  to  develop 
“JCIDS-like”  documents 

Assess-to  Criteria  (AtC)  document 

-  BMDS  Warfighter-developed  requirements 
document 

-  Development  is  on  hold  and  future  is 
unknown 

-  Unlikely  it  would  be  formally  approved  by 
the  JS 
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Decoupled  Testing 


•  DECOUPLED  TESTING: 


-  Each  component/element  test  stands  on  its  own, 
not  dependent  upon  or  being  impacted  by  the 
results  of  other  tests 

-  Identifies  the  separation  of  capability  blocks 
whose  development  shouldn't  depend  on  each 
other 

-  Allows  system  designers  to  have  as  little 
dependencies  as  possible 

-  Reduces  the  risk  of  malfunction  in  one  part  of  a 
system  of  systems  when  other  parts  are  changed 

-  Does  not  need  a  detailed  requirements 
specification/speculation 

•  Need  architecture  diagrams 

•  Need  a  scope  overview 

-  Instead  of  testing  against  the  specification,  the 
independent  testing  effort  will  focus  on: 

•  production-level  system  integration  testing 

•  formal  usability  testing 

-  Supports  DoD  5000.02  concept  of  Integrated  T&E 
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A  Combat  Support  Agency 


IAP  Development 


*  JITC  develops  an  Interoperability 
Assessment  Plan  (IAP)  to  look  at 
System  of  System  BMDS 
interoperability  requirements 

•  Similar  in  concept  to  the  JITC 
Interoperability  Certification 
Evaluation  Plan  (ICEP) 

-  Establishes  a  test  and  evaluation 
strategy  for  evaluating 
interoperability  requirements  in: 

•  the  most  efficient  and  effective 
manner 

•  in  an  operationally-realistic 
environment 

-  Test  and  evaluation  strategy 
identifies: 

•  Data  necessary  to  support  an 
interoperability  evaluation 

•  Test  events/environments  planned 
to  produce  that  data 

-  Certification  vs.  Assessment 
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Elements  of  the  IAP 


A  Combat  Support  Agency 


JITC,  Fort  Huachuca,  AZ 


Element  #5 


Element  #1 


Element  #3 
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A  Combat  Support  Agency 


Element  #1 :  Validate  Overarching 
SoS  Requirements 


•  JCIDS 

•  Capability  Portfolio  Manager 
(CPM) 

•  Communities  of  Interest  (COIs) 

•  Department  of  Defense 
Technology  Standards  and 
Profile  Registry  (DISR) 

•  Concept  of  Operation  (CONOP) 

•  Concept  of  Employment 
(CONEMP) 

•  Joint  Mission  Threads  (JMT) 

•  Meta  Data  Registry  (MDR) 

•  Tactics/Techniques/Procedures 
(TTPs) 

•  IMTP 

•  M&S  Plan 
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Element  #2:  Validate  Elemental 
System  Requirements 


Same  as  SoS  requirements  for 
each  elemental  system  multiplied 
(times  the  number  of  associated 
component  systems) 

Must  identify  commonality  and 
deltas  between  each  element 
requirements  and  the  SoS 
requirements  (times  the  number 
of  associated  component 
systems) 

Must  identify  commonality  and 
deltas  between  each  element 
requirements  and  the 
requirements  of  other  elements 
of  the  SoS  (times  the  number  of 
associated  component  systems) 
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DIS/^ 
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Element  #3:  Develop  T&E  Strategy 
(SoS  and  Elements) 


Test  and  Evaluation  Master  Plan  (TEMP) 

Combined  Test  Team  (CTT)/lnteg rated  Test 
Team/Integrated  Product  Team,  etc 

Development  Test  (DT)  Test  Organization 
documents 

Operational  Test  (OT)  Test  Organization 
documents 

Interoperability  Test  Organization  documents 

Modeling  and  Simulation  Analysis  (Do  models 
and  simulations  exist  which  support  test 
requirements?) 

Major  Range  and  Test  Facility  Base  (MRTFB) 
analysis  (What  existing  infrastructure  exists 
to  support  test  requirements?) 

Analysis  of  emerging  technologies  (What  new 
test  methods  and  equipment  might  be 
required  to  support  test  requirements?) 
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ras/u, 


Element  #4:  Test  Planning/Data 
Collection  Planning  (SoS  and  Elements) 

A  Combat  Support  Agency 


•  Develop  IAP 

-  Standards  Conformance  Testing 
Requirements  and  testing 
organizations 

-  Interoperability  Testing 
Requirements  and  testing 
organizations 

-  OT  Requirements  and  testing 
organizations 

-  Service  Level  Testing  and 
Developmental  Testing 

-  Test  Venues 

•  Laboratory  (Contractor/Government) 

•  Service  Level  Testing  and  DT 

•  JITC 

*  Operational  Test 

*  Combined/Joint  exercises 

*  Post  fielding  assessment  in  the  theater 

-  Types/amount/formats  of  data 

-  Use  of  Design  of  Experiments 

-  Tools  (Data  Collection  &  Analysis) 


13 


DIS/^E 

A  Combat  Support  Agency 


Element  #5:  Analysis  and  Reporting 
(SoS  and  Elements) 


•  Information  Assurance  Results 

•  Spectrum  Results 

•  Standards  Conformance  Testing  Results 

•  Interoperability  Testing  Results 

•  Capabilities  and  Limitations 

•  Data  Analysis  Group  (DAG)/Data  Analysis 
Working  Group  (DAWG) 

•  DT  Results 

•  OT  Results 

•  Net  Readiness  -  Key  Performance 
Parameter  (NR-KPP)  Status 

•  Status  of  Interoperability  Report  (SIR) 

•  Test  Incident  Report  Database 

•  System  Tracking  Program 
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IAP  Data / 

Information  Sources 


•  MDA/Element  Events 

-  FTG 

-  FTD 

-  GTD/GTI 

-  FTT 

-  FTM 

-  FTP 

-  FTO 

-  Exercises/Live  Events 

-  Element  LUTs 


JITC  Events 

-  TDLJIT 

-  DICE 


-  GCN  Testing 


Other  Events 

-  IA  Testing 

-  Juniper  Cobra 


LEGEND: 

DICE  -  DoD  Interoperability  Communications  Exercise 

FTD  -  Flight  Test  Distributed 

FTG  -  Flight  Test  GMD 

FTM  -  Flight  Test  Standard  Missile 

FTO  -  Flight  Test  Operational  Patriot 


FTP  -  Flight  Test  Patriot 

FTT  -  Flight  Test  THAAD 

GCN  -  GMD  Communication  Network 

GTD  -  Ground  Test  Distributed 

GTI  -  Ground  Test  Integrated 


JIT  -  Joint  Interoperability  Test 
LUT  -  Limited  User  Test 
TDL  -  Tactical  Data  Link 
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IAP  Test  Venues 


CMS^ 

A  Combat  Support  Agency 

EHF/FLT4  EHF/FL 

^  « - - 

W  Crossover  ' 


— 

JTIDS  RF  LOS 

— 

EHF / MTJ 

— 

3011  C  TCP /IP 

— 

STJ 

— 

X.25  (422  link) 

o 

NSITE  Passive  Tap 

• 

IES  Passive  Tap 

BOSS 


Definitions: 

Data  Collection  Point  TCP4P 

CD 

Data  Collection  Point  R-SPAN 

® 

Truth  Data 

TENA 

System  Under  Test 

Participant 

Patriot 
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Summary 


A  Combat  Support  Agency 


*  Use  of  decoupled  testing  will  reduce  risk  and 
accelerate  a  Joint  Interoperability  assessment/ 
characterization  of  the  BMDS  System  of  Systems 

*  The  development  of  the  BMDS  IAP  will  enable  JITC 
to  provide  a  more  thorough  assessment  and 
characterization  of  BMDS  capabilities  and 
limitations,  and  provide  the  BMDS  Warfighter  a 
better  understanding  of  the  systems  they  are  using. 
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OKA 

A  Combat  Support  Agency 


Questions? 


Robin  S.  Murray 
Chief,  Tactical  Data  Link  Branch 
Joint  Interoperability  Test  Command 

Robin.Murray@disa.mil 
520-538-5139 
DSN:  879-5139 
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Abstract  ID#:  11710 

Contact  Info: 

LTC  Timothy  Timmons 
520-220-8570 

Joint  Interoperability  Test  Command 
timothv.timmons@disa.mil 


DISCLAIMER 


A  Combat  Support  Agency 


"The  information  provided  in  this  briefing  is  for  general  information 
purposes  only.  It  does  not  constitute  a  commitment  on  behalf  of  the 
United  States  Government  to  provide  any  of  the  capabilities,  systems 
or  equipment  presented  and  in  no  way  obligates  the  United  States 
Government  to  enter  into  any  future  agreements  with  regard  to  the 
same.  The  information  presented  may  not  be  disseminated  without 
the  express  consent  of  the  United  States  Government." 
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Coalition  Interoperability 
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Outline 
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*  Consequences 

*  Current  Working  Solutions 

*  Planning  Tool 

*  Limitations 

*  Future  Way  Ahead 

*  Summary 


Purpose 


A  Combat  Support  Agency 


Considering  the  difficulties  facing  the  US  T&E  community  in 
its  efforts  to  achieve  intra-  and  inter-  service  interoperability, 
the  problems  associated  with  trying  to  tackle  coalition 
interoperability  can  seem  insurmountable. 

This  briefing  defines  the  current  problem,  discusses  current 
ways  US  Combatant  Commands  are  addressing  the  issue, 
and  proposes  a  way  ahead. 


The  Reality 


A  Combat  Support  Agency 


•  US  expects  to  work  in  concert  with  allied 
and  coalition  forces  in  nearly  all  future 
operations 

•  Since  US  participates  in  coalitions  when 
undertaking  both  combat  and  noncombat 
operations,  interoperability  needs  to  be 
addressed  across  the  entire  spectrum  of 
operations 

•  US  will  not  hold  back  on  the  pursuit  and 
acquisition  of  technologically  advanced 
systems 

•  Downside  of  unrestricted  advancement  is 
potential  to  become  a  “technology  island” 


The  Problem 


A  Combat  Support  Agency 


•  Many  complexities  conspire  to  make 
coalition  interoperability  a  difficult  issue 
to  resolve 

•  Consider  the  challenges: 


-  Shrinking  defense  budgets 

-  No  enforcement  mechanism 

-  Various  levels  of  C2  sophistication 

-  Loose  or  nonexistent  international  standards 

-  National  interoperability  first  and  foremost 

-  National  proprietary  equipment 

-  Different  national  requirements  and  priorities 

-  Diverse  procurement  methods 

-  A  constantly  moving  target 

-  Coalition  task  organization  variables 


The  opportunities  to  diverge  versus 
converge  are  great 


»  ^  ■ 
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Consequences  of  Failing 
to  Address  the  Problem 


•  Examples  abound  in  almost  every 
multinational  combat,  peacekeeping, 
humanitarian  assistance  and  disaster  relief 
operation  in  recent  history 

-  Desert  Shield/Desert  Storm 

-  Somalia 

-  Bosnia 

-  Kosovo 

-  Pacific  Tsunami 

-  Haiti  Earthquake 

-  Iraq 

-  Afghanistan 

•  Detrimental  impact  on  mission,  lives  & 
resources 


The  battleground  should  not  be  the 
testing  ground 
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Current  Working  Solutions: 
Endeavor  Workshops 


•  Several  US  COCOMs  are  actively  tackling 
the  issue  of  coalition  interoperability 

•  EUCOM,  AFRICOM,  PACOM  conduct 
annual  workshops 

•  EUCOMs  Combined  Endeavor  exercise  is 
the  oldest,  largest  and  most  sophisticated  of 
the  three 

•  Endeavor  Workshops: 

-A  testing  venue  for  potential  coalition  partners 

-  Comprised  predominantly  of  fielded  systems 

-  Field  assessments  of  interoperability,  not 
laboratory  testing 

-  Identified  interoperability  issues  serve  as  a 
catalyst  for  follow  on  in-depth  testing 

-  Results  thoroughly  documented  and  archived 
to  produce  a  useful  field  guide  for  both  planners 
and  operators 
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Planning  Tool: 
Interoperability  Guides 


Interoperability  Guide  published  at  the 
completion  of  each  Endeavor 

Single  most  important  product  to 
come  out  of  the  exercise 

DVD  media 
Java  based 
Includes: 

-  all  assessments  that  have  been  verified 
and  validated  (912+  at  CE10) 

-  archive  of  all  assessments  since 
workshop  inception 

-  Equipment  Specifications 

Extensive  search  and  query  capability 

Multiple  success  stories  of  guides  use 
by  participants 


■  Interoperability  Guide  Home  -  ?  i?  ^ 

View  C£  21)10  Technical  Summary  Help 
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Limitations  with 
Current  Solutions 


•  Endeavor  programs  are  a  step  in  the 
right  direction  but  they  have  limitations: 

-  Only  identify  interoperability  after  the 
fact 

-  Occurs  only  once  a  year 

-  Brief  test  window 

-  Logistically  intensive 

-  Limited  to  nations  in  a  geographic  area 

-  More  focus  on  equipment  vs.  systems 

-  Limited  scope 

-  No  enforcement  mechanism 

-  Limited  external  synchronization 
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Future  Way  Ahead 


•  Greater  strides  in  coalition 
interoperability  can  and  must  be  made 

•  Discussion  should  start  between 
countries  at  the  highest  international 
strategic  level  to  sort  out  the  competing 
priorities 

•  Greater  strategic  direction  to  working 
groups 

•  Addressing  interoperability  during  the 
development  process 

•  Establishment  of  a  persistent  federated 
on-demand  multinational  test 
environment 

•  Incentives  and  enticements 

•  Expanded  Endeavor  coverage 
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Future  Way  Ahead: 
CIAV,  CTE2 


AMN  Coalition  Interoperability 
Assurance  and  Validation  Working 
Group  is  building  this  model  test 
environment  today 

Coalition  Test  &  Evaluation  Environment 
(CTE2) 

-  Year-round  interoperability  testing  and 
certification  environment 

-  Federation  of  distributed  test  facilities 
based  on  the  CIAV  model 

-  Test  sites  in  US,  UK,  Canada,  Italy  and 
Belgium 

-  Emulates  the  AMN  operational 
environment 

-  Mission  thread  focus 

-  Scope  all  encompassing 

Required  certification  before 
introduction  into  AOR 


CTE2  CIAV  Environment  (Phase  2) 


COP  IM 
COP  LM 
ICC 
IFTS 
IGEOSIT 
JOCWATCH 
CORSOM 
NIRIS 
VoIP 
XMPP 


NC3A  Battlelab 
The  Hague 
NLD 


JITC  Labs  -  Indian  Head  MD 
&  Ft  Huachuca  AZ,  USA 
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C2  Battlelab 
Shrivenham 
GBR 


C2PC 

HeATS 

GrATS 

ICC 

NIRIS 

JADOCS 

XMPP 

VoIP 


JSIC 

USA 


CFBLNet 


CFX-LSL 

Ottawa 

CAN 


CTSF  Battlelab 
Fort  Hood 
TX,  USA 


Battleview  MIP 
UNCLASSIFIED 


GCCS-J 

VoIP 

XMPP 


AFATDS 
AMDWS 
ADSI 
BCS3 
BC  Server 
(PASS) 
C2PC 
CIDNE 
CPOF 
DCGS-A 
FBCB2 
GCCS-A 
JADOCS 
MCS 
MIP 
TAIS 
VoIP 
XMPP. 
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Future  Way  Ahead: 

GCIP,  Predictive  Interoperability 


Global  Communications  Interoperability  Program 

-  A  online  application  with  a  single  interface  to  simultaneously 
references  multiple,  existing  interoperability  databases 

-  Query  data/results  from  all  16,137  field  assessments  (since  1995) 

JITC's  Value  Added 

-  We’re  building  this  unique  capability  “Out-of-Hide” 

-  Leveraging  its  years  of  experience  supporting  COMBINED 
ENDEAVOR,  AFRICA  ENDEAVOR,  and  PACIFIC  ENDEAVOR 
to  affect  the  future... 


-  Databases  are  updated/expanded,  functionality  increased 
with  every  Endeavor  event  JITC  supports 

Impact  to  the  Warfighter 


ft  <VC1P  Mockie 


H»  Ed*  Vtow  Favorite*  Took  Het> 
'it  G  SS  •  Terts  Management 


JITC  Global  Communications  Interoperability  Program 


irtWirtnWtr  - 


-  Within  minutes,  GCIP  provides  quick,  accurate,  concise 
system  interoperability  answers  for  right-now  support 

-  J-6  planners  can  predict  network  and  system 
interoperability — both  good  and  bad — and  plan  accordingly 


.2009  Croatia  HF  SSB  SAILOR 


Specifications  Comments 


-  COCOMs  can  deploy  to  any  theater,  with  almost  any  Nation 
and  be  able  to  predict  what  will  work,  and  what  will  not  work 


HOTLINE 

1-800-LET-JITC 
24/7  Support 


Direct  Intoropere  biltty  Matrix 


Country 


Nomenclature 


Armenia  HARRIS  RF-5800  H  MP  |AM  HARRIS  RF-5800  H  MP)  | 
Bosnia  HARRIS  RF  5022  TESTNET 

Canada  AM/PRC-138  [RT-1694(C)| 

Denmark  ARRIS  RF-5800  H  |DK  ARMY  1|  | 


GCIP  Today 

Contains  system  data  from  over  90  Nations 


Summary 


A  Combat  Support  Agency 

•  Coalition  operations  are  here  to  stay 

•  Coalition  interoperability  has  emerged 
as  a  critical  but  complex  issue,  fraught 
with  great  advantages  and  extremely 
difficult  problems 

•  Ample  opportunities  abound  for 
divergence,  but  convergence  will  take 
determination 

•  COCOM  Endeavor  workshops  are 
tackling  coalition  interoperability  today 

•  The  future  is  a  persistent  federated,  on- 
demand  multinational  test  environment 

•  Most  nations  agree  the  costs  are  worth 
the  headaches 


The  battleground  should  not  be 
_ the  testing  ground^-— — 
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Outline 


•  Purpose 

•  Goals  and  Objectives 

•  DoD  Strategic  Plan  for  T&E  Resources 

•  Processes 

•  Products 


Strategic  Planning  Purpose 


•  Provide  a  vision  of  the  capabilities  and  investments 
needed  to  support  the  testing  of  future  warfighting 
capabilities 

•  Assess  T&E  resources  from  DoD  T&E,  non-DoD 
government,  commercial,  academia,  and 
international 

•  Include  requisite  workforce,  infrastructure,  and 
funding  resulting  in  a  T&E  capability,  by  means  of 
the  T&E  processes 

•  Influence  Service/Agency  T&E  POM  and  Needs  and 
Solutions  investments  through  the  budget 
certification  process 


Goals  and  Objectives 


•  Translate  top-level  (National,  OSD,  &  JCS)  strategic 
guidance  into  strategic  direction  for  the  sustainment, 
improvement,  and  modernization  of  the  DoD  test 
resource  infrastructure 

•  Guide  the  service  &  agency  investment  process 
(Service  POM,  CTEIP,  T&E/S&T,  Reliance) 

•  Assess  the  state  of  the  current  T&E  infrastructure 

•  Provide  a  foundation  for  budget  certification 


Presidential  Guidance,  COCOM  Needs,  DoD  Strategies 

Capabilities/Requirements 

Rapid/Long  Term  Acquisitions 

address  Warfighter  Capabilities 

Test  Requirements 

Test 

Test 

Test 

Resources/Budget 

Infrastructure 

Workforce 

Range  Facilities 
T&E  Resources 

•  Investments 

•  Modernization 

•  Divestments 

•  Sustainment 

•  Operations 


g  ''X 


Service 

Infrastructure 

TCA 


Broad  Scope 
Infrastructure 
TCA 


JIPP 
Investment 
TCA 


* 


Strategic  Plan  provides 
a  comprehensive  and 
traceable  review  of  T&E 
requirements  spanning: 

•  T&E  resources 

•  Facilities 

•  Workforce 

•  Budgets 


SASC  Assessment  of  the 
2010  Strategic  Plan 


“We  view  the  Strategic  Plan  as  an  important  tool  for 
identifying  gaps  in  T&E  resources  and  plans  for 
addressing  them.  We  review  it  with  a  particular  eye  to 
the  grading  system  —  to  evaluate  how  well  POD  is 
meeting  identified  needs  --  and  to  the  identified  gaps 

and  recommended  actions.  We  believe  that  the  report 
is  a  valuable  document  which  should  be  useful  not 
only  to  us,  but  to  the  Department.  The  whole  point  of 
the  strategic  plan  is  to  guide  the  Department's  actions 
with  an  analytic  approach  to  ensure  that  we  have  the 

T&E  resources  that  we  need  in  place  when  we  need 


them. 


6 


-  Congressional  Staff  Member 


T&E  as  an  OSD  Enterprise 


Working  closely  with  other  oversight  groups  helps  TRMC  achieve  its  mission 

and  align  plans/  guidance  overall 


DT&E 

reviews  and  approves, 
the  developmental  tesf 
and  evaluation  plan 
within  the  test  and 
evaluation  master  plan 
for  each  major  defense 
acquisition  program 


ASD  R&E  SE 

works  with  DT&E  to 
ensure  development 
test  and  evaluation 
activities  are  consistent 
with  DoD  processes  for 
systems  engineering  & 
development  plans 


A 


TRMC  SP 

oversees  budgets  for 
the  test  and  evaluation 
facilities  and  resources, 
completes  and 
maintains  the  strategic 
plan,  and  administers 
the  CTEIP  program 


Interlocking 

Oversight 

Tasks 


DOT&E 

guides  SECDEF  on  all 
budgetary  and  financial 
matters  relating  to 
operational  test  and 
evaluation,  including 
operational  test  facilities 
and  equipment 


ASD  R&E 

works  with  DT&E  to 
assess  technological 
maturity  and  integration 
risk  of  MDAP  program 
critical  technologies 
before  Milestone  B 


J 


These  groups  share  guidance 
responsibilities  -  E.g. 

-  Develop  policies  and  guidance 
for  developmental  T&E 
activities  (DT&E) 

-  Guide  and  oversee  operational 
testing  efforts  (DOT&E) 

-  Govern  test  processes  (AS 
R&E  SE) 

-  Provide  insight  into  future 
needs  (ASD  R&E  S&T) 

-  Review  and  guide  expenditures 
for  all  T&E  resources  and 
facilities  within  and  outside  the 
Department  (TRMC) 

-  Review  T&E  requirements  and 
assess  adequacy  to  meet 
requirements  (TRMC) 


Top-Down  Inputs 


Air  Combat 

Cyberspace/CNO/EW/DE 
Land  Combat  (Chem/Bio) 
Sea  Combat 

Space  and  Missile  Defense 


Requirements 

Assessment 

& 

Analysis 


l&M 

Reviews 


TEMPs 

KPPs 


MRTFB 

Annual 

Review 


Service 

Briefs 


Reliance 

Process 


Bottom-up  Inputs 


Test  Capability  Areas 

1 .  Air  Combat 

2.  Land  Combat 

3.  Sea  Combat 
Electronic  Combat 
Space  Combat 
C4ISR 

7.  Armaments  and  Munitions 

8.  Targets  and  Threats 

9.  Test  Environments 

10.  Common  Range  Instrumentation 


Traceability:  Warfighter  Needs 


Warfighter  requirement  needs  testing  ->  back  to  mission  capability 


OSD,  Executive, 
DIA,  S&T 

I 

What  are  the 
emerging  trends  and 
threats? 


Main  External  POC  &  Critical  Questions 


AT&L,  COCOMs 


I 


DT&E,  DOT&E, 
Experts 


What  are  the  capability 
requirements  and 
warfighter  needs? 


T 


SE,  MDAP,  T&E 
Users 


What  tests  are  needed 
given  likely  missions 
and  future  demands? 


X 


What  resources  and 
facilities  are  required  to 
facilitate  T&E? 


Services,  OMB, 
CAPE,  Congress 


I 


What  investments  are 
being  made  to  fulfill 
resource  demands? 


i 


l 


l 


Global  Context  & 
Challenges 


Threats  &  Trends 
Strategic  Objectives 
,  Complex  Operating  , 
Environments 


Warfighter 
Missions  /  Needs  1 


Technology 
Development  (ACTD) 
Acquisition 


,  Current  (available)  , 
vs..  Future  Need 
(for  current 

.  acquisition  programs 
and  future  warfighter 
needs) 


T&E 

Enablers 


Resources 

Infrastructure 

Specialties 

Processes 


Programs  & 
Investments 


T&E  S&T 
Service  POM 
CTEIP 


◄ 


Systematically  Map 
End-to-End 


► 


Partial  /  illustrative  depiction 


Traceability:  Top-Level 
Guidance 


Space  and.  Missile  Defense  Strategic  Objectives 


Reliable,  Affordable,  &  Timely  Access  to  Space 

Protect  Space  Capabilities  from  Interference 

Enable  Defense  and  Intelligence  Operations 

> 

Ensure  Space  Situational  Awareness 

Deny  Enemy  Use  of  Hostile  Capabilities 

j 

Provide  Global  Warning  for  Missile  Defense 

> 

Defend  the  Homeland  and  Allies  vs..  BMD  Attack 

► 

Ensure  BMD  Fiscal  Sustainability  &  Flexibility 

* 

Space 


Missile 

Defense 


10 


Traceability:  Mapping  Needs  to 
Objectives  to  Requirements 


Global  Context  & 
Challenaes 


Threats  &  Trends 
Strategic  Objectives 
Complex  Operating 
Environments 


Manytp.Many 


Warfighter 
Missions  /  Needs 


Technology 
Development  (ACTD) 
Acquisition 


Many  toIVLany 


OcUJcHJIlUV 

Current  (available) 
vs..  Future  Need 
(for  current 
acquisition  programs 
and  future  warfighter 
needs) 


[ 


Reliable,  Affordable,  &  Timely  Access  to  Space 


Protect  Space  Capabilities  from  Interference 


Enable  Defense  and  Intelligence  Operations 


Ensure  Space  Situational  Awareness 


Deny  Enemy  Use  of  Hostile  Capabilities 


Provide  Global  Warning  for  Missile  Defense 


Defend  the  Homeland  and  Allies  vs..  BMD  Attack 


Ensure  BMD  Fiscal  Sustainability  &  Flexibility 


X37  B 


EELV 


GPS  Block 


Minuteman 


NPOESS  /  UPS 


SBIRS  HIGH 


Other 


System  Level  Test  of  ORS  Launch 
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Traceability:  Identify  T&E  Enablers 


T&E 

Capabilit 


,  Current  (available) 
vs..  Future  Need 
(for  current 

.  acquisition  programs 
and  future  warfighter 
needs) 


. Many  to  Magy...  - . 


System  Level  Test  of  ORS  Launch 


Other 


T&E  Enablers 


Resources 

Infrastructure 

Specialties 

Processes 


Mid-Pressure  Arc-Heated  Facility 


New  ORS  Test  Concepts 


Other 


'\ 


Realistic  System  Level  Ground  Test 

Full  Sized  Satellite  Test  Facility 

Secure  Flight  Termination  System  (FTS) 

\ 

Replacement  FTS  Capability 

Forensic  Data  Coll.  Of  Radar  &  Optics 

Satellite  FTS  Capability 

J 


Adequate  Resource; 

Current  resources  fully 
provide  required 
capability. 

Partially  Adequate; 

Current  resources 
partially  provide  required 
capability. 

Inadequate  Resources: 
Current  resources  do  not 
provide  required 
capability. 
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Traceability:  Identify  candidate 
investment  methods 


T&E  Enablers 


Resources 

Infrastructure 

Specialties 

Processes 


. Many  to  ..Many 


Programs  & 
Investments 


T&E  S&T 
Service  POM 
CTEIP 


◄ 


Traceability  example: 
Space  Access 

Systematically  Map  End-to-End 


'  Global  Context  &  i 
Challenaes 


Threats  &  Trends 
Strategic  Objectives 
,  Complex  Operating  J 
Environments 


Warfighter  , 
Missions  /  Needs 


Technology 
Development  (ACTD) 
Acquisition 


T&E 

Caoabilit 


Current  (available)  , 
vs..  Future  Need 
(for  current 
acquisition  programs 
1  and  future  warfighter  1 
needs) 


T&E  Enablers 


Resources 

Infrastructure 

Specialties 

Processes 


Programs  & 
Investments 


T&E  S&T 
Service  POM 
CTEIP 


I 


Reliable,  Affordable,  & 
Timely  Access  to  Space 


Protect  Space 
Capabilities 


Enable  Defense  and 
Intelligence  Operations 


Ensure  Space 
Situational  Awareness 


Deny  Enemy  Use  of 
Hostile  Capabilities 


Provide  Global  Warning 
for  Missile  Defense 


Defend  Homeland  and 
Allies  vs.  BMD  Attack 


Ensure  BMD  Fiscal 
Soundness  &  Flexibility 


X37  B 


EELV 


GPS  Block  I 


System  Level  Test  of 
ORS  Launch 

Realistic  System  Level 
Ground  Test 

Secure  Flight 
Termination  System 


Mid-Pressure  Arc- 
Heated  Facility 


Space  Threat  Assessment 
Testbed  Spiral  1  (CTEIP) 


|_|  $XM 


Full  Sized  Satellite  Test 
Facility 


Replacement  FTS 
Capability 


Minuteman  III 


Forensic  Data  Coll.  Of 
Radar  &  Optics 


NPOESS  /  UPS 


SBIRS  HIGH 


Other 


Other 


DMSP-F15  FTS 
Capability 


New  ORS  Test 
Concepts 


Enhanced  Flight  Termination 
System (CTEIP) 

Joint  Advanced  Missile 
Instrumentation  (CTEIP) 

Subminiature  Flight  Safety 
System  (CTEIP) 


Other 


15 


Traceability:  Warfighter  Needs 


Warfighter  requirement  needs  testing  ->  back  to  mission  capability 


OSD,  Executive, 
DIA,  S&T 

I 

What  are  the 
emerging  trends  and 
threats? 


Main  External  POC  &  Critical  Questions 


AT&L,  COCOMs 


I 


DT&E,  DOT&E, 
Experts 


What  are  the  capability 
requirements  and 
warfighter  needs? 


T 


SE,  MDAP,  T&E 
Users 


What  tests  are  needed 
given  likely  missions 
and  future  demands? 


X 


What  resources  and 
facilities  are  required  to 
facilitate  T&E? 


Services,  OMB, 
CAPE,  Congress 


I 


What  investments  are 
being  made  to  fulfill 
resource  demands? 


i 


l 


l 


Global  Context  & 
Challenges 


Threats  &  Trends 
Strategic  Objectives 
,  Complex  Operating  , 
Environments 


Warfighter 
Missions  /  Needs  1 


Technology 
Development  (ACTD) 
Acquisition 


,  Current  (available)  , 
vs..  Future  Need 
(for  current 

.  acquisition  programs 
and  future  warfighter 
needs) 


T&E 

Enablers 


Resources 

Infrastructure 

Specialties 

Processes 


Programs  & 
Investments 


T&E  S&T 
Service  POM 
CTEIP 


◄ 


Systematically  Map 
End-to-End 


► 


Partial  /  illustrative  depiction 


Products  and  Effects 
of  the  Strategic  Plan 


Strategic  Plan 


T&E  Requirements 
Performance  Measures 
T&E  Facilities  and  Resources  needed 
Investments  needed 
Budgetary  Resources  needed 


Inform  Congress 
for  Appropriations 
and  Legislation 


New  Testing  Challenges 


•  Cyber 

•  Artificial  Intelligence 

•  Autonomy 

•  Nanotechnology 

•  Environments  (GPS-denied,  Domains,  LVC,  etc...) 

•  Space  and  Missile  Defense 

•  Soldier  systems  (medical,  human  behavior/decision¬ 
making) 

•  Directed  Energy 

•  Biometrics 

•  Hypersonics 

•  Energy/Power 


The  Strategic  Plan  and 
Budget  Certification 


•  The  Strategic  Plan: 

-  Is  central  to  the  TRMC  mission  and  associated 
budget  certification  process 

-  Provides  guidance  for  the  planning,  programming, 
and  budgeting  of  T&E  resources 

-  Forecasts  a  period  of  1 0  years 

•  T&E  gaps  identified  in  the  Strategic  Plan  are 
used  during  the  budget  certification  process  to 
ensure  DoD  T&E  capabilities  exist  to  support 
current  and  future  acquisition  programs 
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Strategic  Planning  Products 


•  Biennial  Strategic  Plan  for  DoD  T&E  Resources 

•  Studies  and  Analyses 

-  2010  T&E  Manpower  Conversion  Report  to  Congress  (currently  working) 

-  2010  Airborne  Laser  Study  to  Congress  (TRMC  Member  of  study  group) 

-  2010  Tri-Service  Electronic  Warfare  Test  Capabilities  Study 

-  2009  Joint  Urban  Test  Capabilities  Study  (Lead) 

-  Impact  Report  to  Congress  on  High  Energy  Laser  Systems  Test  Facility  (HELSTF) 
and  Plan  for  T&E  of  High  Energy  Laser  Systems  April  2009 

•  OSD  AT&L  Joint  Analysis  Teams 

-  Space  and  Missile  Defense  Launch  and  Test  (DAB/DAE  Review)  (TRMC/SIO  Leads) 

-  Counter  Improvised  Explosive  Device  (Lead),  2010 

•  Other 

-  Senior  Oversight  Group  for  Nuclear  Weapons  Effects  T&E  (Co-Chair  with  DTRA) 

-  Rapid  Acquisition  Initiatives  (TRMC  and  DDR&E  joint  effort) 

-  OPM  and  OBM  2009  Reform  Initiative  Senior  Working  Group  for  Civilian  Hiring 
(TRMC  Member) 


Note:  DoD  internal  studies  are  not  available  to  public 
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Summary 


•  Provide  a  vision  of  the  capabilities  and  investments 
needed  to  support  the  testing  of  future  warfighting 
capabilities 

•  Foundational  document  that  uses  a  systems 
engineering  process  to  identify  DoD  T&E  resource 
investments  and  gaps 

•  Collaboration  with  OSD,  military  departments,  non- 
DoD  government  Agencies,  and  commercial, 
academia,  and  international  organizations 

•  Influence  Service/Agency  T&E  POM  investments 
through  the  budget  certification  process 
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Contact  Information 

For  a  copy  of  the  2010  Strategic  Plan  for 
T&E  Resources,  please  contact: 

TRMC 

Deputy  Director,  Strategic  Planning 
Dr.  Suzanne  V.  Strohl 
Suzanne.Strohl@osd.mil 


Background 
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Resource  Categories 


Scope: 


r  ^ 

Review  must 
be  specific  to 
each 

acquisition 
AND  specific 
to  each 
range/facility 


r  ^ 

Review 
considers 
capabilities 
inside  and 
outside  of  the 
MRTFB 


■  Investment 

■  Gap  in  existing  Test  Capabilities 

■  New  Test  Capabilities 

■  Modernization 

■  Sustainment 

■  Workforce 

■  Operations 

■  Divestment  (Helps  in  funding  the  above) 


Development  Timeline 


Strategic  Plan 
(even  years) 


Director’s 

Guidance 

Memos 


Drives 


Strategic  Planning 
Infrastructure 
Report 
(odd  years) 


K 

FY  2014 

Drives 

POM 

Annual 

Test 

Infrastructure 

Review 

Requirements 
Reviews  for 
CTEIP  and 
T&E/S&T 


Budget 

Certification 

FY12 


Budget 
Certification 
Review  FY12 


'  ’ 


Director’s 

Guidance 

Memos 


Annual 

Test 

Infrastructure 

Review 


Requirements 
Reviews  for 


CTEIP  and 
T&E/S&T 

Budget 
Certification 
Review  FY13 


yr 


2010 


JAN  MAR  JUNE  JULY  AUG  OCT 


2011 


JAN  MAR  JUNE  JULY  AUG  OCT 


The  STEWARD  of  the  DoD  Test  Enfra structure 

Majo  r  Range  and  Test  Faci  I  ity  Ba  5e  {M  RTF  B):  T  he  VC  ritical  Co  re" 
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Wh  lie  Sand  a  Teat  Center 


Atlantic  Undersea  Teat  and  Evalu  atlon  Center 


fllblfOTING  NATIONAL  SECURITY  SINCE  1919 

_ 


National  Defense  Industrial  Association 


o 

/ 

,  7 

8 

r  -f  { 

O 

Top  Pentagon  leadership 
presentations  on  T&E  / 
acquisition  policy  and  issues 
Industry  leaders  sharing  T&E 
perspectives  and  responses  to 
recent  policy  initiatives 
Special  former  NSA  guest 
speaker  addressing  cyber 
security  policy 
Over  80  speakers  addressing 
a  host  of  issues  facing  today’s 
T&E  community 
Parallel  breakout  sessions 
focused  on  specific  T&E  issues 
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27th  ANNUAL 
NATIONAL  TEST 
&  EVALUATION 
CONFERENCE 

Co-Sponsored  by  the  NDIA  C4ISR  &  Systems 
Engineering  Divisions 

MARRIOTT  TAMPA  WATERSIDE  TAMPA,  FLORIDA 

EVENT  #1910 


TEST  &  EVALUATION  CONFERENCE 
GENERAL  INFORMATION 


CONFERENCE  ANNOUNCEMENT 

The  27th  Annual  National  Test  &  Evaluation  Conference  is  sponsored  by  the  NDIATest 
&  Evaluation  Division  and  supported  by  the  Office  of  the  Under  Secretary  of  Defense 
(AT&T)  and  the  Director,  Operational  Test  &:  Evaluation  (DOT&:E).  Co-sponsors  of  this 
symposium  are  the  C4ISR  and  Systems  Engineering  Divisions  of  NDIA. 

Test  and  Evaluation  is  often  looked  at  by  Program  Managers,  Program  Executive  Officers 
and  other  proponents  of  weapon  systems  as  an  unwelcome  obstacle  to  the  deployment  of 
systems  to  the  Department  of  Defense  and  Homeland  Security.  T&;E  is  often  seen  as  a 
source  of  bad  news  which  can  potentially  delay  the  deployment  of  these  systems  and  add 
to  their  eventual  cost. 

Most  engineers,  technicians  and  program  administrators  recognize  that  test  and  evaluation 
is  an  integral  part  of  the  scientific  method  of  systematically  assessing  the  effectiveness, 
suitability  and  survivability  of  hardware,  software  and  personnel. 

This  national  conference  will  focus  on  policies,  methods,  and  approaches  that  could  better 
serve  the  ultimate  consumer  of  our  T&:E  efforts,  the  Warfighter.  Given  that  Tampa  is  the 
home  of  both  the  U.S.  Special  Operations  Command  and  the  U.S.  Central  Command,  it 
will  provide  a  fertile  opportunity  to  see  and  hear  first-hand  about  how  T&:E  could  better 
serve  our  fighting  forces. 

With  the  recent  combat  surge  into  Afghanistan  and  change  in  our  operational  support  in 
Iraq,  it  is  vital  that  we  take  note  of  the  recent  lessons  learned  in  both  rapid  deployment 
as  well  as  tailoring  our  responses  to  the  changing  environments  and  tactics  our  fighting 
forces  are  now  facing. 

Increasing  fiscal  pressures  also  prompt  us  to  address  T&:E  approaches  to  saving  time  and 
money  as  well  as  to  examine  those  other  disciplines  which  feed  theT&:E  activity,  including 
Systems  Engineering,  Logistics,  C4ISR,  and  R&:D  and  Training. 

Recent  policy  initiatives  will  also  be  addressed  as  to  their  implications,  applications  and 
effectiveness.  Discussions  will  include  how  the  recent  legislative  initiatives  requiring 
additional  T&;E  statutory  responsibilities  for  Developmental  Test  and  Evaluation  are 
being  implemented.  Multiple  topic  tracks  and  tutorial  sessions  will  be  included  in  the 
conference  to  enable  more  focused  discussions  of  specific  topics  enabling  additional  time 
for  Q&A  as  well. 


CONFERENCE  ATTIRE 

Conference  attire  is  business  for  civilians  and  Class  A  uniform  for  military.  In  addition, 
your  identification  badge,  received  upon  conference  check-in,  must  be  worn  at  all  times. 


NDIA  T&E  EXECUTIVE 
BOARD 

Mr.  Joe  Andrese,  AEG  NDIA 
Chapter  * 

Dr.  Suzanne  Beers,  MITRE 
Corporation 

Dr.  Keith  Bradley,  LLNL 

Mr.  Britt  Bray,  DRC 

Corporation 

Mr.  Sam  Campagna,  NDIA 

RADM  David  Crocker,  USN 
(Ret),  Booz  Allen  Hamilton 

Dr.  Paul  Deitz,  AMSAA* 

Mr.  Dick  Dickson,  Tybrin 
Corporation 

Dr.  Anne  Hillegas,  ARA 

Corporation 

Mr.  John  Illgen,  Northrop 
Grumman 

RADM  Bert  Johnston,  USN 

(Ret),  Wyle  Corporation 

Dr.  Mark  Kiemele,  Air  Academy 
Associates 

Mr.  Chuck  Larson,  SURVICE 
Engineering 

Mr.  James  O’Bryon,  The 

OBryon  Group ,  T&E  Division 
Chair 

Mr.  Brendan  Rhatigan, 

Lockheed  Martin 

Mr.  Jack  Sheehan,  ORSA 
Corporation 

Dr.  James  Streilein,  OSD, 
DOT&E* 

Dr.  Lowell  Tonnessen,  IDA 

Dr.  Juan  Vitali,  OSD  CBD * 

Mr.  Martin  Woznica,  Raytheon 
Company 

Mr.  William  Yeakel,  ORSA 
Corporation 

*  Government  liaison  to  NDIA 
T&E  Executive  Board 


TEST  &  EVALUATION  CONFERENCE 
AWARD  INFORMATION 


WALTER  W.  HOLLIS  HONORS  LUNCHEON 

Hie  Walter  W.  Hollis  Award  is  presented  annually  in  recognition  of  lifetime  contributions  and  achievement  in  the  area  of  defense  Test  & 
Evaluation.  The  award  is  presented  in  the  name  of  Walter  W.  Hollis  who  is  recognized  for  his  dedicated  and  long-standing  service  in  the 
field  of  Defense  Test  &  Evaluation.  This  year’s  recipient,  Dr.  James  N.  Walbert,  Chief  Scientist,  SURVICE  Engineering  Company,  will  be 
recognized  at  the  conference  Awards  Luncheon  on  Tuesday,  March  15. 

Previous  Recipients  of  this  Award: 

Dr.  James  J.  Streilein,  Technical  Director/ Deputy  to  the  Commander. ;  U.S.  Army  Test  and  Evaluation  Command  (2010) 

Dr.  Ernest  Seglie,  Science  Advisor  to  the  Director,  Operational  Test  &  Evaluation,  OSD  (2009) 

Dr.  Paul  H.  Deitz,  Technical  Director,  AMSAA,  AEG,  MD  (2008) 

Mr.  James  F.  O’Bryon,  Former  DDOT&E  /  LFT  (2007) 

RADM  Charles  “Bert”  Johnston,  USN  (Ret),  Wyle  Laboratories  (2006) 

Hon  Thomas  Christie,  DOT&E,  OSD  (2005) 

Dr.  Marion  Williams,  HQAFOTEC  (2004) 

Mr.  James  Fasig,  Aberdeen  Test  Center  (2003) 

Mr.  G.  Thomas  Castino,  Underwriters  Laboratories,  Lnc.  (2002) 

Hon  Philip  Coyle,  III,  DOT&E,  OSD  (2001) 

Mr.  Walter  W.  Hollis,  Department  of  the  Army  (2000) 


TESTER  OF  THE  YEAR  AWARDS  LUNCHEON 

These  awards,  presented  to  outstanding  individuals  in  the  field  of  Test  &  Evaluation,  offer  OSD  and  each  Military  Service  Test  &  Evaluation 
Department  the  opportunity  to  select  three  award  recipients  for  recognition  as  the  Tester  of  the  Year  in  specific  categories.  The  three  categories 
recognized  are:  Military,  Civilian,  and  Contractor.  Recipients  will  be  recognized  at  the  conference  Awards  Luncheon  on  Wednesday,  March 
16. 


MAJ  Brian  Spurlock,  USA 

2010 

Army 

Military  Tester  of  the  Year 

Ms.  Patricia  Frounfelker 

2010 

Army 

Civilian  Tester  of  the  Year 

Mr.  Henry  Waller 

2010 

Army 

Contractor  Tester  of  the  Year 

COL  Steven  Duke,  USA 

2010 

OSD 

Military  Tester  of  the  Year 

Ms.  Stephanie  Koch 

2010 

OSD 

Civilian  Tester  of  the  Year 

Mr.  Patrick  Matthews 

2010 

OSD 

Contractor  Tester  of  the  Year 

Maj  Ryan  Voneida,  USAF 

2010 

USAE 

Military  Tester  of  the  Year 

Mr.  William  Nix 

2010 

USAF 

Civilian  Tester  of  the  Year 

Mr.  David  Smith 

2010 

USAE 

Contractor  Tester  of  the  Year 

CDR  John  Verniest,  USN 

2010 

Navy 

Military  Tester  of  the  Year 

Mr.  Don  Nelson 

2010 

Navy 

Civilian  Tester  of  the  Year 

Mr.  Douglas  Cornell 

2010 

Navy 

Contractor  Tester  of  the  Year 

Capt  Todd  Richardson,  USMC 

2010 

Marine  Corps 

Military  Tester  of  the  Year 

Ms.  Cam  Donohue 

2010 

Marine  Corps 

Civilian  Tester  of  the  Year 

Mr.  Eric  Rannenberg 

2010 

Marine  Corps 

Contractor  Tester  of  the  Year 
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MONDAY,  MARCH  14, 2011 


10:00  AM  -  6:00  PM 

CONFERENCE  REGISTRATION  OPEN  -  2ND  LEVEL  REGISTRATION 

10:00  AM  -  2:45  PM 

TUTORIALS  A-D,  SESSION  1  -  SEE  TRACK  LAYOUT  FOR  ROOM  ASSIGNMENTS 

There  is  a  $50  registration  fee  for  tutorial  attendance. 

11 :00  AM -4:00  PM 

DISPLAY  SET-UP  -  GRAND  SALONS  A-D 

12:00  NOON  - 1:00  PM  LUNCH  BREAK 

Lunch  not  included  in  conference  or  tutorial  registration 


2:45  PM  -  3:00  PM 

AFTERNOON  BREAK  -  GRAND  BALLROOM  FOYER 

For  tutorial  registrants  only 

3:00  PM  -  4:30  PM 

TUTORIALS  E-H,  SESSION  2  -  SEE  TRACK  LAYOUT  FOR  ROOM  ASSIGNMENTS 

4:30  PM 

TUTORIALS  CONCLUDE 

5:00  PM  -  6:00  PM 

6:00  PM 

KICKOFF  RECEPTION  IN  THE  DISPLAY  AREA  -  GRAND  SALONS  A-D 

Open  to  all  conference  registrants 

CONFERENCE  ADJOURNED  FOR  THE  DAY 

6:00  PM 
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MONDAY,  MARCH  14,  2011  —  Tutorials 


10:00  AM -2:45  PM 


TUTORIAL 

TUTORIAL  A 

Session  Chair:  Dr.  Paul  Deitz, 
AMSAA 

Grand  Salon  G 

TUTORIAL  B 

Session  Chair:  Mr.  Martin  Woznica, 
Raytheon  Company 

Grand  Salon  H 

TUTORIAL  C 

Session  Chair:  Dr.  Suzanne  Beers, 
MITRE  Corporation 

Grand  Salon  I 

TUTORIAL  D 

Session  Chair:  Mr.  Britt  Bray,  DRS 
Corporation 

Grand  Salon  J 

SESSION  1 

SESSION  1 

SESSION  1 

SESSION  1 

10:00  AM 

1 1678  -  Using  DFSS  as  an 

Integrating  Framework  for  MBT&E 
and  DOT&E 

Dr.  Mark  Kiemele,  President  and  Co¬ 
founder,  Air  Academy  Associates 

1 1694  -  Efficient  Modeling  and 
Simulation  (M&S)  Using  Design  of 
Experiments  (DOE)  Methods 

Dr.  Tom  Donnelly,  Principal 

Customer  Advocate,  Systems  Engineer, 
JMP 

11653  -  Test  Planning  —  Advancing 
the  Science 

Mr.  Steve  Scukanec,  Senior  Test 
Engineer,  Northrop  Grumman 

Aerospace  Sector 

1 1488  -  Testing  and  Evaluating 
Intranets,  Portals,  and  Enterprise 
Systems  for  Usability 

Dr.  Patricia  Chalmers,  Chief  Science 
Advisor,  U.S.  Joint  Forces  Command 

SESSION  1  CONTINUED 

SESSION  1  CONTINUED 

SESSION  1  CONTINUED 

SESSION  1  CONTINUED 

1:00  PM 

1 1678  -  Using  DFSS  as  an 

Integrating  Framework  for  MBT&E 
and  DOT&E 

Dr.  Mark  Kiemele,  President  and  Co¬ 
founder,  Air  Academy  Associates 

1 1694  -  Efficient  Modeling  and 
Simulation  (M&S)  Using  Design  of 
Experiments  (DOE)  Methods 

Dr.  Tom  Donnelly,  Principal 

Customer  Advocate,  Systems  Engineer, 
JMP 

11653  -  Test  Planning  —  Advancing 
the  Science 

Mr.  Steve  Scukanec,  Senior  Test 
Engineer,  Northrop  Grumman 

Aerospace  Sector 

1 1488  -  Testing  and  Evaluating 
Intranets,  Portals,  and  Enterprise 
Systems  for  Usability 

Dr.  Patricia  Chalmers,  Chief  Science 
Advisor,  U.S.  Joint  Forces  Command 

3:00  PM  -  4:30  PM 


TUTORIAL 

TUTORIAL  E 

Session  Chair:  Dr.  Lowell  Tonnessen, 
IDA 

Grand  Salon  G 

TUTORIAL  F 

Session  Chair:  Mr.  Dick  Dickson, 
Tybrin  Corporation 

Grand  Salon  H 

TUTORIAL  G 

Session  Chair:  Mr.  Chuck  Larson, 
SURVICE  Engineering 

Grand  Salon  I 

TUTORIAL  H 

Session  Chair:  Mr.  Brendon 
Rhatigan,  Lockheed  Martin 

Grand  Salon  J 

SESSION  2 

SESSION  2 

SESSION  2 

SESSION  2 

1 1703  -  Ships  Are  Different 

Mr.  Mark  Lucas,  Command  Technical 

11693  -  Modern  Design  of 
Experiments  (DOE)  Methods 

11570  -  A  Day  in  the  Life  of  a 
Verification  Statement 

11705  -  Defense  Information 

Systems  Agency  Joint 

Interoperability  Test  Command 

3:00  PM 

Director,  Combat  Direction  Systems 
Activity 

Dr.  Tom  Donnelly,  Principal 

Customer  Advocate,  Systems  Engineer, 
JMP 

Mr.  Steve  Scukanec,  Senior  Test 
Engineer,  Northrop  Grumman 

Aerospace  Sector 

Interoperability  Support  for  the 
Afghanistan  Mission  Network 

Mr.  Jeffery  Phipps,  CIAV  Co-Chair, 

US  Lead,  JITC 
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TUTORIAL  A:  USING  DFSS  AS  AN  INTEGRATING 
FRAMEWORK  FOR  MBT&E  AND  DOT&E 

This  tutorial  will  provide  attendees  a  comprehensive  process 
to  capture  all  of  the  activities  in  MBT&E  and  DOT&E 
needed  to  achieve  a  successful  system  acquisition.  It  will  use 
DFSS  in  its  more  expansive  connotation,  namely  Designing 
for  Successful  Systems  vice  Design  for  Six  Sigma,  the  more 
common  but  limited  meaning.  DFSS  starts  with  the  voice 
of  the  warfighter  (or  customer)  and  the  required  operational 
capability.  These  requirements  are  then  flowed  down  to  the 
critical  performance  measures  using  tools  that  help  to  prioritize 
along  the  way.  The  performance  measures  may  include  KPPs, 
MOEs,  MOSs,  and  CTPs.  The  critical  performance  measures 
are  linked  to  key  design  parameters,  and  once  this  linkage  is 
firm,  performance  optimization  can  be  accomplished.  Design 
of  Experiments  (DOE)  is  shown  to  be  a  critical  player  in  the 
design  and  optimization  phases,  as  well  as  in  every  facet  of 
testing  and  evaluation.  Once  the  design  and  performance  is 
optimized,  it  must  be  validated  and  the  capability  rolled  back 
up  to  the  system  level  capability.  DFSS  will  be  shown  as  an 
interdisciplinary  activity,  spanning  the  activities  of  systems 
engineering,  reliability  engineering,  design  and  optimization, 
test  and  evaluation,  and  system  capability  confirmation. 

TUTORIAL  B:  EFFICIENT  MODELING  AND  SIMULATION 
(M&S)  USING  DESIGN  OF  EXPERIMENTS  (DOE) 
METHODS 

Attendees  will  learn  how  Design  of  Experiments  (DOE) 
methods  can  be  used  to  extract  the  most  useful  information 
from  computer  simulation  models.  They  will  see  how  the 
sequential  running  of  blocks  of  simulations  can  be  used  to 
conduct  the  overall  fewest  trials  necessary  to  do  sensitivity 
analysis  of  the  factors  being  studied.  They  will  also  see  how 
to  develop  a  fast-running  (seconds)  surrogate  model  —  which 
testers  and  analysts  can  interactively  query  —  of  a  long- 
running  (hours,  days  or  weeks)  simulation.  Design  solutions 
will  include  the  application  of  traditional  DOE  methods  to 
discrete  event  and  agent-based  simulations,  and  modern  space¬ 
filling  designs  to  more  complex  physics-based  simulations 
such  as  Computational  Fluid  Dynamics  (CFD).  When  to 
use,  and  how  to  choose  between  traditional  linear  regression 
approximation  methods  and  spatial  regression  interpolation 
methods  will  be  discussed.  The  effective  practice  of  using 
checkpoint  simulations  for  determining  the  accuracy  of 
surrogate  model  predictions  will  be  demonstrated. 


TUTORIAL  C:  TEST  PLANNING  —  ADVANCING  THE 
SCIENCE 

Test  planning  is  rapidly  becoming  a  lost  art.  Many  test 
planning  activities  are  based  solely  on  corporate  knowledge  and 
“Like  we  did  it  last  time”  theories.  Solidifying  requirements 
development,  improving  the  programs  verification  and 
validation  activities,  increased  program  collaboration  and 
streamlined  test  programs  are  all  benefits  of  a  solid  and 
well  defined  test  planning  approach.  By  increasing  program 
collaboration  and  the  overall  time  spent  on  the  “engineering 
of  a  program”  while  significantly  reducing  the  time  required 
producing  the  engineering  verification  and  validation  artifacts, 
solid  model  based  test  planning  can  ensure  that  a  test  program 
is  more  effective  across  its  lifecycle.  This  tutorial  examines  the 
test  planning  process.  From  verification  to  test  plan  modeling 
and  test  plan  generation,  participants  will  see  the  processes  and 
tool  sets  in  action.  To  demonstrate  some  of  these  capabilities, 
participants  will  generate  test  requirements  and  objectives, 
model  the  plan,  optimize  the  plan  and  assign  resources, 
and  finally  generate  a  simple  test  plan  while  maintaining 
connections  to  the  original  requirements  intent. 

TUTORIAL  D:  TESTING  AND  EVALUATING  INTRANETS, 
PORTALS,  AND  ENTERPRISE  SYSTEMS  FOR 
USABILITY 

This  tutorial  will  teach  attendees  how  to  perform  intranet, 
portal,  and  enterprise  usability  evaluations.  Attendees  are 
encouraged  to  come  with  a  project  in  mind  as  they  will  be 
worked  on  throughout  the  tutorial.  Attendees  will  learn 
how  to  analyze  their  stakeholders’  goals  and  needs:  How  to 
decide  who  their  stakeholders  are,  decide  which  stakeholders 
to  include  in  their  evaluation,  choose  a  random  sample  of  end 
users,  and  determine  stakeholders’  goals/needs.  Attendees  will 
learn  how  to  design  a  Usability  Evaluation:  How  to  budget 
time,  knowing  what  types  of  T&E  methods  are  possible, 
deciding  what  methods  to  use,  designing  a  first-rate  survey, 
determining  sample  completion  tasks,  deciding  how  many 
methods  to  use,  and  how  to  quantify  usability  data.  Attendees 
will  write  a  design  for  their  portal  evaluation  including  topics 
discussed.  Information  will  be  provided  on  How  to  Evaluate 
Your  Portal  Usability  Evaluation:  Pilot  evaluations,  participant 
performance,  survey  understandability,  task  understandability, 
determining  if  tasks  are  too  easy  or  too  hard,  understanding 
the  data,  feedback  from  participants,  making  improvements. 
Attendees  will  also  learn  how  to  write  their  reports.  Portal 
evaluation  samples  will  be  provided. 
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TUTORIAL  E:  SHIPS  ARE  DIFFERENT 

Recent  fleet  concerns  with  surface  ship  and  system 
performance  have  punctuated  the  need  to  evolve  the 
Navy’s  ship  T&E  processes  and  practices  in  such  a  way  that 
enables  acquisition  decisions  that  are  based  on  a  framework 
of  mission  area  effectiveness  and  suitability.  However, 
because  any  given  ship  supports  multiple  missions  through 
the  employment  of  a  complex  array  of  systems,  sensors, 
and  weapons,  the  aforementioned  changes  truly  require 
a  “system  of  systems”  approach.  This  approach  must  take 
care  in  balancing  multiple  systems  at  differing  states  of 
lifecycle  maturity  through  their  development  processes.  This 
necessitates  a  progressive  examination  of  systems  maturity 
using  mission-based,  measureable,  testable  artifacts.  This 
tutorial  will  discuss  the  Navy’s  Mission  Based  Test  Design 
methodology  and  illustrate  how  its  application  through  an 
Integrated  Test  process  can  be  used  in  ship  and  ship  systems 
acquisition.  It  will  also  discuss  how  this  approach  can  enable 
improved  rigor  leading  to  a  better  understanding  of  risks 
and  warfighting  effects,  thereby  facilitating  the  information 
quality  needed  for  effective  ship  deployment  decisions. 

TUTORIAL  F:  MODERN  DESIGN  OF  EXPERIMENTS 
(DOE)  METHODS 

This  tutorial  will  provide  attendees  the  very  latest 
experimental  designs  published  since  2008.  References  will 
be  provided  for  four  new  types  of  design  that  offer  testers  the 
ability  to  run  either  fewer  trials  or  for  the  same  number  of 
trials,  learn  more  about  interactions  or  quadratic  behavior. 
These  recently  peer-reviewed  designs  have  not  yet  made 
it  into  textbooks.  The  new  designs  include  non-regular 
orthogonal  fractional-factorial,  robust  screening,  alias- 
optimal,  and  Bayesian  D-optimal  supersaturated  designs. 
Comparisons  between  these  new  alternative  methods 
and  traditional  designs  will  be  provided  to  show  the  new 
methods  are  superior  or  strong  competitors. 

TUTORIAL  G:  A  DAY  IN  THE  LIFE  OF  A  VERIFICATION 
STATEMENT 

One  measure  of  the  quality  of  a  product  requirement  is  that  it 
be  verifiable.  Verifiability  assessment  is  one  of  the  exit  criteria 
for  the  Systems  Requirements  Review  and  is  necessary  for 
requirement  validity.  Nomination  of  one  or  more  verification 
methods  (examination,  analysis,  demonstration  or  test)  is 
often  taken  as  the  sole  evidence  of  verifiability.  A  completed 
Verification  Cross  Reference  Matrix  is  frequently  considered 


as  the  final  verifiability  assessment  and  responsibility  for 
the  remainder  of  the  verification  effort  is  transferred  to  the 
test  and  evaluation  and  other  implementing  communities 
for  completion.  Lessons  learned  from  many  programs  have 
shown  that  a  more  robust  application  of  systems  engineering 
should  include  the  requirements  engineers  (with  detailed 
knowledge  of  product  requirement  intent)  working  with 
the  verification  implementing  organizations  as  the  best 
combination  to  define  the  verification  requirements.  Such 
definition  should  include  statement  of  the  verification 
objectives,  success  criteria  and  environment.  Including 
this  information  in  the  ’’Quality  Assurance”  section  of  the 
requirements  document  allows  for  buy-in  by  the  customer 
well  in  advance  of  implementing  the  verification  activities. 
This  information  is  used  by  verification  personnel  to 
generate  one  or  more  verification  plans  and  to  develop  the 
detailed  verification  program.  Verification  requirements 
are  planned  into  verification  events  which  are  executed 
using  the  proper  system  elements  and  environments.  These 
verification  requirements  are  key  to  establishing  long  lead 
verification  facilities,  tools  and  laboratories.  Early  definition 
of  these  requirements  helps  prevent  facility  re-designs  and 
verification  re-plans  that  can  cause  expensive  delays.  Finally, 
verification  data  analysis  is  performed,  and  the  information 
compiled  into  verification  reports  certifying  system  product 
requirements  compliance.  This  robust  verification  approach 
will  provide  proof  of  requirements  satisfaction,  leading  to 
systems  that  meet  the  customers’  needs  at  a  lower  life-cycle 
cost.  This  presentation  explores  the  value  of  well-crafted 
verification  requirements  developed  early  in  the  Program. 
A  “Day  in  the  Life  of  a  Verification  Requirement”  shows 
the  interaction  and  benefits  of  verification  requirements  to 
the  verification  execution  teams.  The  presentation  will  offer 
a  lifecycle  description  of  the  verification  requirement  from 
conception  to  certification. 
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TUTORIAL  H:  DEFENSE  INFORMATION  SYSTEMS  AGENCY  JOINT  INTEROPERABILITY  TEST  COMMAND 
INTEROPERABILITY  SUPPORT  FOR  THE  AFGHANISTAN  MISSION  NETWORK 

USCENTCOM  operates  in  a  coalition  environment  and  must  be  able  to  generate  and  pass  critical  information  to  U.S.  and 
coalition  partners.  The  Command  and  NATO,  as  members  of  the  International  Security  Assistance  Forces  (ISAF),  understand 
that  widespread  interoperability  is  a  key  component  to  achieve  effective  and  efficient  operations.  These  communication 
capabilities  must  include  a  wide  variety  of  not  only  military  governmental  operations,  but  also  non-governmental  agencies 
and  industrial  partners.  To  that  end,  they’ve  created  the  Afghan  Mission  Network  (AMN)  and  commissioned  the  Defense 
Information  Systems  Agency’s  Joint  Interoperability  Test  Command  to  develop  the  Coalition  Test  and  Evaluation  Environment 
(CTE2)  testing  arm  of  the  Coalition  Interoperability  Assurance  and  Validation  (CIAV)  process.  The  AMN  is  the  backbone  or 
core  infrastructure  that  will  provide  long-term  communications  and  information  system  and  satellite  communication  services 
to  support  the  ISAF  as  it  expands  its  operations  across  the  country  during  the  ongoing  operations.  This  tutorial  will  discuss 
the  eight  core  critical  Coalition  Mission  threads,  phases  for  testing,  and  how  the  JITC  stood  up  a  network  and  is  testing  the 
systems  in  a  distributed  hardware  in  the  loop  environment  to  ensure  interoperability  across  the  AMN.  It  will  also  discuss  the 
applicability  to  other  theaters  that  may  need  to  implement  a  similar  process. 


TUESDAY,  MARCH  15, 2011 


7:00  AM  -  6:30  PM 
7:00  AM  -  8:00  AM 
8:00  AM 

8:05  AM 


CONFERENCE  REGISTRATION  OPEN  -  2ND  LEVEL  REGISTRATION 
CONTINENTAL  BREAKFAST  IN  THE  DISPLAY  AREA  -  GRAND  SALONS  A-D 
OPENING  REMARKS  -  GRAND  SALONS  E-F 

►  Mr.  Sam  Campagna,  Assistant  Vice  President,  Operations,  NDIA 

TRIBUTE  TO  OUR  NATION  AND  WARFIGHTERS,  NATIONAL  ANTHEM 


SESSION  A:  CONFERENCE  WELCOME  &  KEYNOTES 

8:10  AM  WELCOME  AND  CONFERENCE  INTRODUCTORY  REMARKS 

►  Mr.  James  O’Bryon,  Chairman,  NDIA  T&E  Division;  The  O’Bryon  Group 

8:20  AM  CONFERENCE  KEYNOTE  ADDRESS 

►  Honorable  Dr.  J.  Michael  Gilmore,  Director,  Operational  Test  &  Evaluation,  OSD 
Honorable  Dr.  J.  Michael  Gilmore  was  sworn  in  as  Director  of  Operational  Test  and  Evaluation  on  September 
23,  2009.  A  Presidential  appointee  confirmed  by  the  United  States  Senate,  he  serves  as  the  senior  advisor  to  the 
Secretary  of  Defense  on  operational  and  live  fire  test  and  evaluation  of  Department  of  Defense  weapon  systems. 

Prior  to  his  current  appointment,  he  was  the  Assistant  Director  for  National  Security  at  the  Congressional 
Budget  Office  (CBO),  and  was  responsible  for  CBO’s  National  Security  Division.  Dr.  Gilmore  is  a  former 
Deputy  Director  of  General  Purpose  Programs  within  the  Office  of  the  Secretary  of  Defense,  Program  Analysis 
and  Evaluation  (OSD(PA&E)).  Dr.  Gilmore  also  has  served  as  the  Division  Director  of  Operations  Analysis 
and  Procurement  Planning,  within  the  Office  of  the  Deputy  Director,  Resource  Analysis  and  as  an  Analyst  for 
Strategic  Defensive  and  Space  Programs  Division,  Office  of  the  Deputy  Director,  Strategic  and  Space  Programs. 

Dr.  Gilmore’s  service  with  Program  Analysis  and  Evaluation  covered  1 1  years.  Early  in  his  career,  Dr.  Gilmore 
worked  at  the  LLNL,  Livermore,  California  performing  research  in  their  magnetic  fusion  energy  program.  He 
has  also  worked  with  Falcon  Associates,  McLean,  VA,  and  the  McDonnell  Douglas  Washington  Studies  and 
Analysis  Group.  Dr.  Gilmore  is  a  graduate  of  MIT  where  he  earned  a  B.S.  in  Physics.  He  subsequently  earned  a 
M.S.  and  Ph.D.  in  Nuclear  Engineering  from  the  University  of  Wisconsin. 
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9:00  AM  GUEST  SPEAKER 


►  Honorable  Frank  Kendall,  Principal  Deputy  Under  Secretary  of  Defense,  AT&L,  OSD 


Mr.  Frank  Kendall  was  sworn  in  as  Principal  Deputy  Under  Secretary  of  Defense  for  Acquisition, 
Technology,  and  Logistics  (PDUSD(AT&L))  on  March  5,  2010.  In  his  role  as  PDUSD(AT&L),  Mr. 
Kendall  is  authorized  to  act  for  and  provide  assistance  to  the  Under  Secretary  of  Defense  for  Acquisition, 
Technology  &  Logistics  (USD  (AT&L)).  He  also  advises  and  assists  the  USD  (AT&L)  in  providing  staff 
advice  and  assistance  to  the  Secretary  of  Defense  on  the  acquisition  system;  research  and  development; 
modeling  and  simulation;  systems  engineering;  advanced  technology  and  developmental  test  and  evaluation. 
Within  government,  Mr.  Kendall  held  the  position  of  Director  of  Tactical  Warfare  Programs  in  the  Office  of 
the  Secretary  of  Defense  and  the  position  of  Assistant  Deputy  Under  Secretary  of  Defense  for  Strategic  Defense 
Systems.  Mr.  Kendall  was  also  Vice  President  of  Engineering  for  Raytheon  Company.  Mr.  Kendall  also  spent 
ten  years  on  active  duty  with  the  Army  serving  in  Germany,  teaching  Engineering  at  West  Point,  and  holding 
research  and  development  positions.  He  is  a  Distinguished  Graduate  of  the  U.S.  Military  Academy  at  West  Point 
and  he  holds  a  Masters  Degree  in  Aerospace  Engineering  from  California  Institute  of  Technology,  a  Master  of 
Business  Administration  degree  from  C.W.  Post  Center  of  Long  Island  University,  and  a  Juris  Doctoris  from 
Georgetown  University  Law  Center. 


9:30  AM 


HOMELAND  SECURITY  T&E  PERSPECTIVES 

►  Mr.  Gary  Carter,  Director,  Test  &  Evaluation  and  Standards  Division,  Department  of 
Homeland  Security 


10:00  AM  MORNING  BREAK  AND  NETWORKING  IN  THE  DISPLAY  AREA  -  GRAND  SALONS  A-D 


SESSION  B:  OTA’S  (OPERATIONAL  TEST  AGENCY’S)  ROUNDTABLE 

Session  B  Chair  and  Roundtable  Moderator:  Dr.  Catherine  Warner,  Science  Advisor,  DOT&E,  OSD 

10:30  AM  ROUNDTABLE 

►  MG  Genaro  Dellarocco,  USA,  Commander,  ATEC 

►  RADM  David  Dunaway,  USN,  Commander,  OPTEVFOR 

►  Maj  Gen  David  Eichhorn,  USAF,  Commander,  AFOTEC 

►  Col  David  Reeves,  USMC,  Commander,  MCOTEA 

►  COL  Joseph  Puett,  USA,  Commander,  JITC 

1 1 :30  AM  WALTER  W.  HOLLIS  HONORS  LUNCHEON:  PRESENTATION  FOR  OUTSTANDING  LIFETIME 

ACHIEVEMENT  IN  DEFENSE  TEST  &  EVALUATION  -  FLORIDA  SALONS  l-IV 

►  Dr.  James  N.  Walbert,  Chief  Scientist,  SURVICE  Engineering  Company 

Dr.  Walbert  has  more  than  35  years  of  DoD  T&E  and  related  experience  including  extensive  and  novel  work 
as  an  interior  and  exterior  ballistician,  a  vulnerability/lethality  tester  and  analyst,  a  materials  engineer,  and  an 
author  and  instructor.  From  1974  to  1978,  Dr.  Walbert  served  as  a  mathematician  and  test  director  for  the 
U.S.  Army  Material  Testing  Directorate,  where  he  planned,  analyzed,  evaluated,  and  assessed  a  wide  range 
of  engineering  test  programs.  From  1978  to  2000,  he  served  as  a  research  scientist/engineer  for  the  Ballistic 
Research  Laboratory  (and  then  the  Army  Research  Laboratory)  and  from  2001  to  2003,  Dr.  Walbert  served  as 
Chief  Scientist  for  the  DARPA  Future  Combat  Systems  Program  Office.  Since  joining  SURVICE  in  2003  as 
the  Chief  Scientist,  Dr.  Walbert  has  developed  numerous  analytical  processes  for  exploitation  of  ballistic  test 
data.  He  has  authored/co-authored  more  than  50  technical  publications  during  his  career,  including  the  AIAA- 
published  text  Fundamentals  of  Ground  Combat  System  Ballistic  Vulnerability/Lethality,  which  was  named  ARL’s 
Publication  of  the  Year  for  2009.  Based  on  this  text,  Dr.  Walbert  also  developed  and  teaches  a  highly  acclaimed 
basic  ballistic  vulnerability  course  to  Government  and  industry  practitioners  throughout  the  T&E  community. 

Dr.  Walbert  holds  a  B.S.,  M.S.,  and  Ph.D.  in  mathematics  all  from  the  University  of  Delaware. 
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1 1 :30  AM  LUNCHEON  GUEST  SPEAKER:  SOME  PROBLEMS  OF  CYBER  SECURITY 


►  Mr.  Robert  L.  Deitz,  former  General  Counsel ,  National  Security  Agency 
Robert  L.  Deitz  is  currently  Distinguished  Visiting  Professor  &  CIA  Officer-in-Residence  at  George  Mason 
University.  From  2006  until  February  2009  he  served  as  Senior  Councillor  to  the  Director  of  the  Central 
Intelligence  Agency.  From  September  1998  to  September  2006  he  was  the  General  Counsel  at  the  National 
Security  Agency  where  he  represented  the  NS  A  in  all  legal  matters.  Fie  has  also  held  positions  as  Acting  General 
Counsel  at  the  National  Geospatial-Intelligence  Agency  and  as  Acting  Deputy  General  Counsel,  Intelligence,  at 
the  Department  of  Defense.  Professor  Deitz  began  his  career  as  a  law  clerk  to  the  Fionorable  Justices  Douglas, 
Stewart,  and  White  of  the  United  States  Supreme  Court.  He  has  also  been  in  private  practice  and  was  Special 
Assistant  to  Deputy  Secretary  of  State  Warren  Christopher  and  to  Secretary  of  Health,  Education  and  Welfare 
Joseph  Califano  during  the  Carter  Administration.  Professor  Deitz  received  his  J.D.  (magna  cum  laude)  from 
Harvard  Law  School,  where  he  was  the  Supreme  Court  Note  and  Note  Editor  of  the  Harvard  Law  Review.  He 
received  an  M.P.A.  from  the  Woodrow  Wilson  School  of  Public  and  International  Affairs  at  Princeton  University, 
where  he  studied  international  politics  and  economics.  He  majored  in  English  literature  at  Middlebury  College 
where  he  received  a  B.A.  (cum  laude)  and  became  a  member  of  Phi  Beta  Kappa. 


SESSION  C:  ACQUISITION  REFORM  -  THE  IMPACT  ON  INDUSTRY 

Session  C  Chair:  Dr.  Suzanne  Beers,  MITRE  Corporation 

1 :1 5  PM  PENTAGON  RESPONSE  TO  CONGRESSIONAL  STRENGTHENING  OF  DT&E 

►  Mr.  Chris  DiPetto,  Principal  Deputy,  Developmental  Test  &  Evaluation 

1 :45  PM  REPORT  ON  NDIA’S  INDUSTRIAL  COMMITTEE  ON  TEST  &  EVALUATION  (IC0TE) 

►  Mr.  James  Ruma,  Chairman. ,  NDIA  ICOTE;  Vice  President ,  Engineering,  GDIS 
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5:30  PM  -  6:30  PM  RECEPTION  IN  THE  DISPLAY  AREA  -  GRAND  SALONS  A-D 
6:30  PM  CONFERENCE  ADJOURNED  FOR  THE  DAY 


WEDNESDAY,  MARCH  16, 2011 


7:00  AM  -  5:25  PM 
7:00  AM  -  8:00  AM 
8:00  AM 


CONFERENCE  REGISTRATION  OPEN  -  2ND  LEVEL  REGISTRATION 
CONTINENTAL  BREAKFAST  IN  THE  DISPLAY  AREA  -  GRAND  SALONS  A-D 
INTRODUCTION  AND  OPENING  REMARKS  -  GRAND  SALONS  E-F 

►  Mr.  Sam  Campagna,  Assistant  Vice  President ,  Operations ,  NDIA 
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SESSION  L:  A  RE-ENERGIZED  DT&E 

Session  L  Chair:  Mr.  John  Illgen,  Chairman, ,  NDIA  National  Board;  Northrop  Grumman 

8:05  AM  PANEL:  T&E:  SERVING  THE  WARFIGHTER  IN  A  COST-CONSTRAINED  ENVIRONMENT 

Panel  Moderator: 

►  Mr.  Chris  DiPetto,  Principal  Deputy,  Developmental  Test  &  Evaluation 
Panelists: 

►  Mr.  David  K.  Grimm,  Acting  Director,  Deputy  Under  Secretary  of  the  Army,  T&E  Office 

►  Mr.  Steve  Hutchison,  DISA  T&E  Executive 

►  Mr.  John  Manclark,  Air  Force  T&E  Executive 

►  Ms.  Amy  Markowich,  Navy  T&E  Executive 

►  Mr.  Tom  Wissink,  Director  of  Integration,  T&E,  Lockheed  Martin 

9:00  AM  SPECIAL  GUEST  PRESENTATION: 

EVALUATION  OF  THE  SINKING  OF  THE  CHEONAN  KOREAN  NAVAL  SHIP 

►  MG  Jong  Sung  Yoon,  Republic  of  Korea  Army  (Ret),  Leader  of  the  International  Investigation 
Team 

Rarely  does  one  have  the  opportunity  to  fully  investigate  the  circumstances  leading  up  to  the  attack  on  and 
sinking  of  a  warship  and  then  be  able  to  recover  the  ship  and  perform  an  extensive  international  investigation 
of  the  threat,  the  damage  and  casualties,  the  computer  modeling  of  the  damage  and  assessment  of  the  causes  and 
effects.  MG  Yoon  led  the  international  investigation  team  of  which  the  US  was  an  integral  part  into  the  sinking 
of  the  Republic  of  Korea’s  warship,  the  CHEONAN,  this  past  year.  His  insights  should  be  instructive  and  of  great 
interest  to  the  conference  attendees.  It  is  a  privilege  to  welcome  him  to  be  a  special  part  of  our  conference  this  year. 

In  addition,  MG  Yoon  will  be  joined  by  Dr.  Young  Shin,  Professor,  Naval  Postgraduate  School  and  visiting 
Professor,  Korean  Advanced  Institute  for  Science  and  Technology,  to  discuss  the  efforts  of  the  International 
Investigation  Team  addressing  the  CHEONAN  sinking. 

MG  Jong-Sung  Yoon  was  born  on  April  4th,  1975  in  Inje-gun,  Gangwon-do,  Korea.  In  1981,  he  received  his  B.S. 
from  the  Korea  Military  Academy  (37th);  in  1999,  MG  Yoon  received  his  M.S.  in  Science  of  public  administration 
from  Dongguk  University;  in  2008,  he  received  his  Ph.D.  in  Politics  from  Myongji  University. 

10:00  AM  MORNING  BREAK  AND  NETWORKING  IN  THE  DISPLAY  AREA  -  GRAND  SALONS  A-D 

SESSION  M:  RESPONSIVE  AND  AGILE  INFORMATION  SYSTEMS  T&E  PANEL 

Session  M  Chair  and  Panel  Moderator:  Dr.  Steve  Kimmel,  Chairman,  NDIA  C4ISR  Division;  Senior  Vice  President, 

Alion  Science  &  Technology 

10:30  AM  PANEL 

Panelists: 

►  Dr.  Steven  Hutchison,  Director  T&E,  DISA 

►  Dr.  James  Streilein,  Deputy  Director,  Net-Centric  and  Space  Systems,  DOT&E 

►  Ms.  Darleen  Mosser-Kerner,  Deputy  Director,  Capabilities  Development,  Office  of  the  Director, 
DT&E 

►  Mr.  Eustace  King,  Chief,  Acquisition  and  Technology,  DOD-CIO/NII 
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1 1 :30  AM  LUNCHEON  -  TESTER  OF  THE  YEAR  AWARDS  -  FLORIDA  SALONS  l-IV 

This  awards  event  is  a  highlight  of  our  annual  conference  since  it  provides  the  opportunity  to 
recognize  outstanding  achievement  in  test  and  evaluation  by  members  of  our  armed  forces,  DoD 
civilians  and  DoD  contractors.  Furthermore,  what  makes  these  awards  particularly  noteworthy  is 
that  the  selections  are  made  by  the  organizations  of  those  being  recognized.  Congratulations  to  all 
who  are  being  recognized  for  their  2010  accomplishments. 

SESSION  N:  IMPROVING  THE  T&E  PROCESS 

Session  N  Chair:  Dr.  Lowell  Tonnessen,  IDA 

1 :1 5  PM  T&E  AND  MISSION  ASSURANCE 

►  Mr.  James  W.  Wade,  Vice  President ,  Raytheon  Company 

1 :45  PM  S0C0M  T&E  PERSPECTIVES:  SERVING  THE  WARFIGHTER 

►  LTC  Kevin  Vanyo,  USA,  USSOCOM J8-0 

►  Mr.  Robert  D.  Werner,  Jr.,  Senior  Test  Officer ;  USSOCOM J8-0 

2:1 5  PM  -  5:25  PM  CONCURRENT  SESSIONS  0  -  V 
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SESSION  O 

T&E  M&S  for  Specific 
Applications 

Grand  Salon  G 

RADM  Bert  Johnston, 
USN  (Ret),  Wyle 
Corporation 

1 1560  -  A  Comprehensive 

Approach  to  Characterizing 
the  Hazards  of  Explosive 
Countermeasures  With  Respect  to 
Dismounted  Troops 

Mr.  Stephen  Swann,  U.S.  Army 
Research  Laboratory 

1 1 529  -  Expanding  Use  of  the 
Probability  of  Raid  Annihilation 
(PRA)  Test  Bed  Across  the  Ship 
Self-Defense  Enterprise 

Mr.  Richard  Lawrence,  AVW 
Technologies 

SESSION  P 
Approaches  to 
Organizing  an 
Effective  M&S 
Program  in  Support 
of  T&E 

Grand  Salon  H 

Mr.  Britt  Bray,  DRS 
Corporation 

1 1500  -  Modeling  and  Simulation 
for  Mission-Based  Test  and 
Evaluation  (MBT&E) 

Mrs.  Beth  Ward,  U.S.  Army 

Research  Laboratory 

1 1476  -  A  Paradigm  for  Modeling  and  Simulation  in  support  of  Mission- 
Based  Test  and  Evaluation 

Dr.  James  Walbert,  SURVICE  Engineering  Company 

SESSION  Q 
T&E 

Instrumentation 

Infrastructure 

-  Maximum 

Utilization  of 
Available  Resources 

Grand  Salon  I 

Mr.  Dick  Dickson, 
Tybrin  Corporation 

1 1497  -  Joint  Mission 

Environment  Test  Capability 
(JMETC):  Improving  Distributed 
Capabilities 

Mr.  Chip  Ferguson,  JMETC 

11508  -  U.S.N.  RDTE  Project 
Support  Aircraft 

Mr.  Charles  Myers,  U.S.  Navy, 
NAWCAD 

1 1 626  -  Dugway  Proving  Ground 
as  the  MRTFB  Chem  Bio  Activity 

Ms.  Jean  Baker,  U.S.  Army  Dugway 
Proving  Ground 

SESSION  R 
Applications 
of  Design  of 
Experiments  (DoE) 
to  T&E 

Grand  Salon  J 

Mr.  Martin  Woznica, 
Raytheon  Company 

1 1677  -  Using  Design  of 
Experiments  (DoE)  to  Integrate 
Developmental  and  Operational 
T&E 

Dr.  Mark  Kiemele,  Air  Academy 
Associates 

1 1 549  -  Probability  Driven 
Experiments  Design  for 

Autonomous  Systems 

Mr.  Troy  Jones,  Charles  Stark 

Draper  Laboratory 

11532  -  Design  of  Experiments: 
Managing  Expectations 

Mr.  James  Carpenter,  AVW 
Technologies,  Lnc. 
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MODULE  1:  INTRODUCTION 

First,  a  few  important  notes  before  we  begin:  If  you  have  questions,  please  feel  free  to  ask  them 
as  we  go  through  this  tutorial  or,  if  you  prefer,  write  your  questions  on  page  22  and  we  can 
address  them  at  the  end  of  class  if  we  have  time.  Also  please  write  any  feedback  on  page  23.  I 
take  all  comments  seriously  and  I  appreciate  any  and  all  feedback. 

Introduction  to  Attendees.  For  our  coursework  today,  first  we  will  introduce  ourselves.  Please 
tell  the  audience  your  name,  a  little  background  about  yourself,  why  you  are  here  and  if  you  can, 
think  of  a  usability  test/evaluation  to  which  you  would  like  to  apply  this  tutorial.  Your 
introductions  will  help  me  know  which  modules  to  stress  to  accommodate  your  needs.  This 
introduction  will  also  help  you  identify  colleagues  who  may  have  backgrounds  or  interests 
similar  to  your  own.  You  may  want  to  network  with  each  other  during  our  break  or  during  the 
conference  this  week. 

Activity:  Let’s  introduce  ourselves  to  the  rest  of  the  attendees,  and,  again,  try  to  mention  one 
project  to  which  you  could  apply  this  tutorial. 


NOTES 

Discussion.  I  usually  have  people  from  many  backgrounds  and  with  varied  levels  of  experience. 
I’ll  try  to  accommodate  you  all  in  my  presentation,  but  please  stop  me  or  slow  me  down  if  there 
is  anything  you  do  not  understand. 

Introduction  to  Coursework.  In  order  to  clarify  the  process  of  measuring  usability  and  the 
process  of  choosing  which  method  or  methods  for  measuring,  I’ll  first  define  a  few  usability 
terms:  usability,  usability  metrics,  test,  evaluation,  stakeholder,  and  user  (Module  2).  Next,  we 
will  review  reasons  for  using  usability  metrics  and  applications  for  this  tutorial  (Module  3),  and 
why  it’s  important  to  apply  what  you  learn  today  (Module  4).  Next  we’ll  talk  about  Stakeholder 
Goals  and  Requirements  (Module  5).  Next,  the  tutorial  steps  through  the  process  of  deciding 
what  portal  usability  methods  to  use  (Module  6)  as  well  as  when,  where,  and  how  to  use  them 
(Module  7).  Then,  we  will  evaluate  our  metrics  (Module  8),  and  the  methods  we  used  to  obtain 
those  metrics  (Module  9).  And  finally,  I’ll  walk  you  through  a  sample  portal  evaluation  report 
(Module  10). 
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MODULE  2:  DEFINITIONS  AND  APPLICATIONS 

For  the  purpose  of  this  tutorial,  I  define  usability  as  user  effectiveness,  efficiency,  and 
satisfaction;  and  usability  metrics  as  the  measures,  in  numbers,  of  user  effectiveness,  efficiency, 
and  satisfaction.  Also,  I  will  use  the  Defense  Acquisition  University’s  definition  of  test  to 
include  a  procedure  designed  to  obtain,  verify,  or  provide  data  for  an  evaluation;  and  the  Defense 
Acquisition  University’s  definition  of  evaluation  as  the  process  for  review  and  analysis  of  all 
data  obtained  from  a  design  review,  hardware  inspection,  Modeling  and  Simulation  (M&S), 
testing,  or  operational  usage  of  equipment.  To  understand  the  difference  between  test  and 
evaluation,  think  of  evaluation  as  the  big  umbrella  -  and  sometimes  testing  is  included  in  the 
evaluation  (or  under  the  umbrella),  and  sometimes  testing  is  not  included.  Also,  for  this  tutorial, 
the  term,  stakeholder  includes  anyone  affected  by  your  test/evaluation.  Of  all  these  definitions,  I 
take  the  definition  of  “stakeholder”  most  seriously,  because  if  I  don’t  consider  all  my 
stakeholders  in  my  test/evaluation,  my  results  may  not  be  credible,  my  recommendations  may 
not  be  followed,  or,  people  who  feel  marginalized  may  be  tempted  to  be  uncooperative.  I’ve 
seen  a  few  definitions  of  the  word  user:  One  definition  equates  the  end  user  with  the  user.  One 
equates  the  user  with  the  customer.  For  this  tutorial,  when  I  speak  of  the  user,  I  mean  the  end 
user. 

Psychology  experiments  show  that  if  we  can  apply  course  material  to  our  lives,  we  learn  that 
course  material  much  more  easily  than  if  we  cannot  apply  it.  We  also  know  that  presenting  our 
ideas  in  front  of  colleagues  can  help  us  learn  and  remember  material.  And  finally,  we  know  that 
presenting  our  ideas  in  front  of  colleagues  can  also  help  us  see  other  points  of  view  and  add  to 
our  knowledge  immensely.  In  fact,  conferences  such  as  this  one  are  a  perfect  example  of  a 
forum  for  learning,  open  discussion,  questions,  and  even  criticism  of  theories,  ideas,  and 
concepts. 

Activity:  So,  with  that  in  mind,  think  of  a  test/evaluation  project  you  are  going  to  apply  this 
tutorial  to,  and  then  enter  it  into  the  box  on  page  5.  It  can  be  a  project  you’ve  conducted  in  the 
past  or  are  conducting  now.  But  it  can  also  be  a  project  you  will  be  doing  in  the  future,  or  would 
like  to  do  in  the  future.  If  you  cannot  discuss  your  project, 


NOTES 


Usability  Test/Evaluations  to  Which  I  Would  Like  to  Apply  This 
Tutorial: 
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MODULE  3:  THE  IMPORTANCE  OF  STAKEHOLDERS 


In  Module  2  we  talked  about  the  definition  of  stakeholders.  That  definition  included  anyone 
affected  by  your  evaluation.  In  the  box  below  I’ve  written  some  ideas  to  start  us  thinking  about 
who  are  stakeholders  are. 

Activity:  Now,  for  the  project  you  brought  to  this  tutorial,  the  one  you  wrote  on  page  5,  circle 
the  stakeholders  (e.g.,  present  end  users,  potential  end  users,  designers,  developers,  funders, 
senior  staff,  and  help  desk  personnel)  who  you  think  you  may  affect  by  your  usability 
test/evaluation  project).  Then  write  in  the  space  next  to  the  ones  you  circled,  who  those  people 
are,  if  you  know  their  names.  If  you  don’t  know  their  names,  write  down  who  you  would  contact 
to  find  out  who  those  people  are. 

I  purposely  did  not  include  all  types  of  stakeholder  groups,  because  I  was  hoping  you  could  think 
of  a  few  more.  There  is  a  space  at  the  bottom  of  the  list  for  any  additional  stakeholders  you  may 
think  of  who  I  do  not  have  listed. 


If  you  are  finished  early,  take  a  look  at  the  discussion  questions  to  prepare  for  a  discussion  of  this 
topic. _ 


NOTES 

Present  End  Users: 

Potential  End  Users: 

Designers: 

Developers: 

Funders: 

Senior  Staff 

Help  Desk  Personnel: 

Additional 

Stakeholders: 

Discussion  Questions: 

Who  are  your  end  users  or  potential  end  users? 

Do  you  agree  that  end  users,  funders,  senior  staff,  are  help  desk  personnel  are 
stakeholders?  Can  you  think  of  any  other  stakeholders? 

Do  you  see  any  problem  with  involving  these  people? 

Do  you  see  any  problems  if  you  don’t  involve  certain  people? 
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MODULE  4:  THE  IMPORTANCE  OF  METRICS 

Potential  customers  usually  want  to  know  how  they  will  benefit  if  they  conduct  a  usability 
test/evaluation,  and  they  may  want  numbers  to  measure  those  benefits,  or  calculate  a  return  on 
investment.  Do  you  know  what  a  return  on  investment  is?  It  is  the  value  (usability  value  for  this 
tutorial’s  purposes),  in  monetary  terms:  In  this  case,  the  value  of  the  test/evaluation  you  are 
about  to  plan.  However,  many  professionals  become  confused  when  customers  ask  them  to 
measure  usability  value,  in  monetary  terms  -  if  any  of  you  are  confused  about  how  to  measure 
usability  value,  we’ll  cover  that  also.  During  Modules  5-10  I’ll  walk  you  through  your  project, 
explain  steps  to  using  good  metrics,  the  methods  to  obtain  those  metrics,  evaluating  your  metrics 
and  methods,  and  reporting  your  metrics.  First,  though,  I’d  like  to  briefly  talk  about  the  reasons 
that  good  metrics  are  important  in  a  test  and  evaluation. 

Reasons  to  employ  portal  usability  metrics  include,  but  are  not  limited  to: 

1 .  Assisting  stakeholders  in  understanding  the  need  for  portal  usability 

2.  Calculating  a  return  on  investment  for  former  or  potential  users 

3.  Helping  your  stakeholders  easily  visualize  portal  usability  findings  (for  example,  via 

graphs,  pie  charts,  and  bar  charts) 

4.  Clearly  conveying  the  results  of  usability  test/evaluations  to  stakeholders 

5.  Advancing  the  field  of  usability 

6.  Evaluating  yourself-  to  realize  if  you  have  served  your  stakeholders 

7.  Helping  your  boss  evaluate  you 

8.  Winning  awards  © 


Activity:  Note  if  any  of  the  eight  sample  motives  I  just  mentioned  (and  repeated  in  the  box 
below)  apply  to  your  project.  Write  down  any  additional  motives  that  apply  to  your  project. 


Motivations  to  Use  Metrics 

Applies  to  My 

Project? 

1.  Assist  stakeholders  to  understand  the  need  for  portal 
usability 

2.  Calculate  a  return  on  investment 

3.  Help  stakeholders  visualize  findings  (via  graphs  and  charts) 

4.  Clearly  convey  results 

5.  Advance  the  field  of  usability 

6.  Evaluate  myself 

7.  Help  my  boss  evaluate  me 

8.  Award 

Other  Motivators? 

Discussion  Questions: 

Which  of  the  motivators  applied  or  did  not  apply  to  your  work?  Why  or  why  not? 
What  additional  motivators  did  you  identify? 
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MODULE  5:  DETERMINE  YOUR  METRICS  BY  ANALYZING 
GOALS/REQUIREMENTS 

In  previous  modules  we  talked  about  who  we  are  and  why  we  are  here,  I  defined  terms,  we 
talked  about  applying  this  tutorial  to  your  projects,  and  the  importance  of  metrics.  Now  we  can 
start  our  tests  and  evaluations.  But,  Where  to  start?. ..How  to  start?  I  always  start  by  analyzing 
my  stakeholders’  goals/requirements  because  those  requirements  will  determine  my  metrics  and 
the  metrics  will  determine  my  methods.  For  example,  if  the  customer  wants  decreased  time  to 
perform  tasks,  my  metrics  should  include  time  to  complete  representative  tasks.  If  end  users 
want  to  experience  certain  emotions  (such  as  satisfaction  with  the  system,  trust  in  the  system)  my 
metrics  should  measure  those  emotions.  If  customers  want  end  users  to  remember  on-line 
training,  I  will  need  to  measure  users’  recall  of  the  training  material.  Here  are  some  more 
examples  of  how  to  determine  your  metrics:  Note  that  I  always  use  stakeholder 
goals/requirements  to  determine  metrics. 


CUSTOMER  REQUIREMENTS  AND  EXAMPLES  OF  METRICS  TO  MEASURE 

THOSE  REQUIREMENTS 

1)  For  a  Redesigned  Portal:  A  customer’s  goal  may  be  to  increase  collaboration  within  the 
organization.  In  this  case,  you  might  measure  the  number  of  cross  division  collaborations  after 
portal  improvements  compared  to  the  number  of  collaborations  before  portal  improvements 

2)  For  a  Data  Repository:  A  customer’s  goal  may  be  to  enable  users  to  find  information 
efficiently.  In  this  case,  you  may  tally  the  number  of  minutes  it  takes  users  to  find  information  in 
the  data  repository. 

3)  For  a  Portal  Training  Program:  A  customer’s  goal  may  include  passable  scores  on  tests 
following  an  on-line  training  course.  You  may  conduct  tests  immediately  after  on-line  training 
to  measure  short-term  recall,  as  well  as  at  longer  intervals  after  training  to  measure  long-term 
retention. 

4)  For  a  Portal  Training  Program:  A  customer’s  goal  may  be  to  ensure  user  satisfaction  with 
the  on-line  training.  In  this  case,  you  may  develop  a  survey  to  measure  self-reported  satisfaction. 

5)  For  a  Portal  Command  and  Control  Program:  If  a  customer’s  goal  is  to  decrease  errors  by 
command  and  control  operators,  you  can  measure  the  number  of  errors,  fatal  errors,  and/or 
successes  in  accomplishing  tasks  in  a  simulation  situation,  or  in  an  actual  situation. 

EXAMPLES  OF  METRICS 


Numerical  Metrics 

•  Number  of  keystrokes  to  perform  a  task 


•  Number  of  clicks  to  reach  information 


•  Number  of  positive  responses  to  the  portal 

•  Number  of  negative  responses  to  the  portal 

•  Number  of  help  queries 

•  Number  correct  on  a  test 


Economic  Metrics 


•  Estimated  return  on  investment 

•  Actual  return  on  investment 
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Time  Metrics 

•  Time  to  complete  a  task. 

•  Time  to  Train. 

•  Time  to  Reach  Fatigue 

•  Latency  time  to  upload  information 

Percentage  Metrics 

•  Percent  of  users  who  score  “very  satisfied,”  “somewhat  satisfied,”  “somewhat 
dissatisfied,”  ...  on  a  survey  before  usability  improvements. 

•  Percent  of  users  who  scored  “very  satisfied,”  “somewhat  satisfied,”  “  somewhat 
dissatisfied,”  ...  on  a  survey  after  usability  improvements 

•  Percent  of  participants  who  noted  boredom 

•  “  satisfaction 

•  “  ease  of  use 

•  Percent  of  users  who  return  to  the  portal  to  find  new  information 

•  Percent  of  users  who  are  successful  in  making  new  collaborations 

•  Percent  of  pages  that  comply  with  usability  guidelines 

•  Percent  of  participants  who  make  errors 

•  Percent  of  participants  who  make  fatal  errors 

•  Percent  of  participants  who  successfully  complete  all  tasks 

Activity:  For  the  usability  test/evaluation  project  you  wrote  on  page  5,  write  in  the  space  below 
a  few  sample  goals/requirements  of  your  customers  (or  potential  customers)  and  other 
stakeholders.  Try  to  choose  goals/requirements  that  you  anticipate  would  be  the  most 
challenging  to  measure.  Then,  opposite  each  goal,  write  the  metrics  that  would  show  if  the 
portal  you  are  testing/evaluating  meets  those  goals/requirements. 


STAKEHOLDER  GOALS/REQUIREMENTS 

METRICS  TO  MEASURE 
IF  THE  SOFTWARE 
MEETS  THOSE  GOALS 

Discussion  Questions: 

What  are  a  few  of  your  customers’  goals/requirements? 

Using  those  goals/requirements,  what  are  your  metrics? 

If  you  have  trouble  determining  metrics  for  your  customer’s  goals  what  are  they? 
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MODULE  6:  DETERMINE  YOUR  TEST/EVALUATION  METHODS 

When  analyzing  their  test/evaluation  projects,  usability  professionals  determine:  Who  will 
participate  in  the  usability  test/evaluation  project,  What  you  will  test/evaluate,  When  you  will 
test/evaluate,  Where  you  will  test/evaluate,  and  How  you  will  test/evaluate. 

Activity:  Return  to  page  9  and  note  a  few  of  the  stakeholder  goals/requirements  you  need  to 
meet.  Enter  them  below  under  the  first  column.  Then,  answer  questions  in  the  next  columns  for 
each  of  your  requirements. 


Goal  or 
Req’ment 
and  Metric 

to  Measure 

Who  will  I 
measure? 

Novices? 

Experts? 

Actual  Users? 
Represent  “? 

Number  of 
users? 

What  will  I 

measure? 

Simulated 

Tasks? 

Real  Tasks? 

Adherence 

to 

Guidelines? 

When  will  I 
measure? 

Before 

Learning? 

After 

Learning? 

Formative? 

Interim? 

Summative? 

Where 
will  I 
measure? 

At  user’s 

work 

station? 

In  a  lab? 

How  will  I 
measure? 

Objective 

test? 

Subjective 
test  (survey)? 

Observation? 

Cognitive 

Walk- 

Through? 

Comparative 

Experiment? 

Data 

Collection? 

Technical 

Trackers? 

How 
will  I 
recruit 
? 

Discussion  Questions:  Share  with  the  audience  one  or  your  usability  test/evaluation  projects, 
along  with  your  metrics,  and  methods.  What  challenges  do  you  anticipate  facing? 
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MODULE  7:  IMPLEMENT  YOUR  METHODS 
A  Sample  of  Guidelines  for  Implementing  Your  Methods: 

1 .  Before  starting,  determine  if  your  method  will  fit  into  your  schedule  and  budget. 

2.  Before  starting,  consult  with  colleagues  to  get  their  opinion  of  your  methods. 

3.  Before  starting,  discuss  your  metrics  &  methods  with  your  team. 

4.  Consider  a  pilot  test/evaluation.  The  beauty  of  doing  a  small  preliminary 

test/evaluation  is  that  you  can  determine  if  you  have  problems  with  your  method  before 
doing  your  test/evaluation  project. 

5.  Use  appropriate  numbers  of  participants  for  an  experiment  (30  in  each  group),  use  proper 
statistical  tests,  note  the  p  value  (the  percent  chance  of  error)  in  your  results,  &  draw 
participants  from  a  random  sample  that  is  representative  of  typical  end  users. 

6.  Be  consistent  with  all  participants. 

7.  In-brief  participants;  provide  a  consent  form;  provide  out-brief. 

8.  Demonstrate  impartiality  to  the  product  so  you  do  not  influence  the  participants. 

9.  Remind  participants:  You  are  not  testing/evaluating  them,  you  are  testing/evaluating  the 
portal 

10.  Read  from  a  script  when  giving  instructions 

11.  Note  variations  in  types  of  participants:  Gender,  Age,  Experience  Levels,  Comfort 
Levels,  Fatigue  Levels,  Intelligence  Levels,  and  Education  Levels  -  anything  that 

may  skew  results. 

12.  For  a  cognitive  walk-through: 

Capture  all  user  comments  while  remaining  impartial  to  the  design  and  comments 
Follow  up  on  nuances  (furrowed  brow,  nervous  speech,  sighs...) 

13.  For  a  behavioral  observation: 

Establish  inter-rater  reliability  if  more  than  one  observer. 

Establish  intra-rater  reliability. 

Consider  audio/video  technology  (for  example  Morae  technology). 

Remain  inconspicuous,  but  available  in  case  of  problems. 

14.  For  a  heuristic  evaluation  (how  well  a  portal  design  follows  usability  guidelines): 

Choose  heuristics  for  which  research  shows  effectiveness. 

If  you  look  at  all  screens,  you  can  then  make  statements  such  as,  “75%  of 
pages  on  the  portal  complied  with  the  guideline  to  provide  consistency 
of  look  and  feel.” 

15.  For  tests:  Test  on  tasks  that  are  representative  of  end  users’  tasks. 

16.  For  surveys:  Use  an  appropriate  number  of  points  (1-5  or,  better  yet,  1-7). 

Discussion  Questions: 

Do  you  see  any  problems  with  any  of  the  above  methodology  guidelines? 

If  costs  and/or  time  were  constrained,  would  you  eliminate  or  reduce  effort  on  any  of  the 
above  methodology  guidelines? 

This  is  simply  a  sample  of  methodology  guidelines.  Can  you  think  any  more  guidelines? 
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MODULE  8:  EVALUATE  YOUR  PROJECT’S  USABILITY  METRICS 

Please  note  that  up  to  this  point,  when  I  used  the  word  “evaluation,”  I  used  it  to  describe 
evaluating  a  portal.  However,  in  this  module  and  the  next  one,  I  speak  of  evaluation  in  terms  of 
evaluating  yourself;  in  other  words,  in  terms  of  evaluating  your  own  usability  project:  your 
metrics,  your  method/s. 


A  Few  Guidelines  for  Evaluating  Your  Metrics 

Evaluate  iteratively  -  this  means  you  evaluate  yourself  throughout  the  analysis,  design, 
development,  and  implementation  of  your  test/evaluation. 

Consult  with  a  mentor  or  your  peers  to  get  their  opinions  of  your  metrics.  Be  open  to  their 
critiques. 

Consult  peer-reviewed  literature  to  keep  up  on  metric  guidelines,  their  effectiveness,  and  the 
advantages  and  disadvantages  of  using  certain  metrics. 

Attend  local  usability  meetings,  national  conferences,  and  international  conferences  to  present 
your  ideas  and  to  obtain  critiques. 

Activity: 

Review  the  metrics  you  wrote  on  page  10  under  the  column  “What  Will  I  Measure?”  and  ask 
yourself: 

Were  my  metrics  appropriate? 

Do  I  need  additional  metrics? 

Will  my  stakeholders  be  satisfied  with  the  metrics  I  obtained?  Why  or  why  not? 


NOTES 

Discussion  Questions: 

Do  you  understand  the  difference  between  evaluating  a  portal  and  evaluating  yourself? 
What  are  some  pitfalls  to  be  aware  of  when  you  evaluate  your  own  metrics? 

What  are  some  advantages  to  evaluating  your  metrics? 
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MODULE  9:  EVALUATE  YOUR  USABILITY  PROJECT  METHODS 

In  addition  to  evaluating  your  metrics,  as  you  did  in  the  previous  module,  you  should  also 
evaluate  your  methods. 

The  next  time  you  conduct  a  usability  test/evaluation,  go  to  page  1 1  and  carefully  evaluate  your 
methodologies  against  those  guidelines,  as  well  as  any  other  guidelines  developed  here  and 
elsewhere. 

As  with  metrics,  the  following  apply  to  methods: 

Evaluate  iteratively  -  this  means  you  evaluate  your  method  throughout  the  analysis,  design, 
development,  and  implementation  of  your  test/evaluation. 

Consult  with  a  mentor  or  your  peers  to  get  their  opinions  of  your  methods.  Be  open  to  their 
critiques. 

Consult  peer-reviewed  literature  to  keep  up  on  usability  methodology  guidelines,  their 
effectiveness,  their  advantages,  and  disadvantages. 

Attend  local  usability  meetings,  national  conferences,  and  international  conferences  to  present 
your  ideas  and  obtain  critiques. 

Activity:  Review  page  1 1  and  note  below,  on  page  13,  if  you  did  or  did  not  follow  those 
guidelines,  as  well  any  other  guidelines  we  developed  as  a  class,  or  that  you  obtain  from  peer- 
reviewed  literature  and  conferences. 


NOTES 

Discussion  Questions: 

What  are  some  pitfalls  to  be  aware  of  when  you  evaluate  your  own  methods? 
What  are  some  advantages  to  evaluating  your  methods? 


14 


MODULE  10:  REPORT  YOUR  TEST/EVALUATION  FINDINGS 
Guidelines  for  Reporting  Your  Results: 

1)  In  addition  to  simply  verbalizing  problems,  I  prefer  to  use  screen  captures  for  illustration.  It 
is  helpful  to  note  the  exact  pages  where  problems  occur. 

2)  In  addition  to  pointing  out  usability  problems,  also  include  usability  successes  -  no  one 
wants  to  just  hear  a  list  of  criticisms.  I  have  never  seen  a  usability  test/evaluation  project  that  has 
absolutely  no  qualities.  In  addition,  if  customers  do  not  know  what  is  working,  they  may 
inadvertently  delete  a  feature  that  is  actually  working  for  them.  Note  the  successes  in  the 
Conclusion  section  of  the  sample  Evaluation  Report  on  pages  16-17. 

3)  Note  also  that  I  removed  all  identifiers  from  the  evaluation  report. 

4)  Pointing  out  not  only  problems,  but  also  offering  recommendations,  is  very  helpful  to 
customers  who  may  not  know  about  usability  and  how  to  make  it  better.  See  examples  in 
Appendix  A,  under  the  “Recommendations”  section  on  page  17. 

5)  Using  bar  charts  and  tables  are  great  ways  of  capturing  the  overall  essence  of  a  report  in 
picture-form.  See  examples  in  Appendix  B  on  page  21. 

6)  See  the  information  provided  in  the  tutorial  if  you  are  doing  a  usability  heuristic  evaluation 
(an  evaluation  of  a  portal  against  usability  guidelines).  This  example  of  one  guideline  is 
especially  useful  to  customers  because  it  not  only  references  the  guideline,  but  also  makes  useful 
comments.  In  addition,  the  example  gives  the  “strength  of  importance”  to  most  end  users,  the 
“strength  of  evidence”  supporting  the  guideline,  and  the  sources  to  back  up  the  guideline.  Please 
see  “Research-Based  Web  Design  and  Usability”  by  Sanjay  J.  Koyani,  Robert  W.  Bailer,  and 
Janice  R.  Nall  for  more  great  examples.  Their  work  explains  the  “Strength  of  Importance”  and 
the  “Strength  of  Evidence”  scales  and  gives  many  more  usability  guidelines  and  examples.  Also, 
you  may  be  able  to  see  more  at  http: //usability.  20  v/pdfs/suidelines.  html. 
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APPENDIX  A 

USABILITY  HEURISTIC,  OR  GUIDELINE,  CHECKLIST  & 
USABILITY  EVALUATION  REPORT 

Following  is  an  example  of  a  heuristic  evaluation.  I  selected  heuristics  that  my  stakeholders 
deemed  important  and  also  heuristics  that  I  deemed  important  over  years  of  consulting.  I  then 
looked  at  a  representative  sample  of  portal  pages  to  determine  if  the  portal  adhered  to  the 
heuristics.  Below  is  a  sample  checklist  followed  by  conclusions,  recommendations,  and  a  note 
of  thanks  on  page  19  for  the  opportunity  to  conduct  the  test/evaluation. 

USABILITY  CHECKLIST 

Visibility  of  System  Status: 

1 .  Does  the  portal  enable  users  to  know  where  they  are  within  the  portal? 

2.  Does  the  portal  enable  users  to  know  where  they  are  in  a  process? 

3.  Does  the  portal  enable  users  to  know  how  much  time  a  process  will  take? 

4.  Does  the  portal  provide  the  user  with  a  way  to  go  back  to  topics  without  repeatedly 

keying  the  Back  button? 

Consistency 

1 .  Does  the  portal  use  consistent  terminology? 

2.  Does  the  portal  use  consistent  color? 

3.  Does  the  portal  use  consistent  layout? 

4.  Does  the  portal  use  consistent  font? 

5.  Does  the  portal  use  consistent  input  and  output  formats? 

Speaking  the  User’s  Language: 

1 .  Does  the  portal  use  natural  language  (not  jargon,  acronyms,  or  system  terms) 

2.  If  stakeholders  insist  on  jargon,  acronyms,  and  system  terms,  is  there  a  glossary? 
Giving  the  User  Control  and  Freedom 

1 .  Does  the  portal  provide  shortcuts  for  frequent  users? 

2.  Does  the  portal  support  Undo/Redo  commands 

3.  Does  the  interface  make  it  difficult  to  perform  irreversible  actions? 

Error  Prevention  and  Error  Recovery 

1 .  Are  potential  errors  noted? 

2.  When  errors  occur,  can  the  user  see  the  source  of  the  error? 

3.  When  errors  occur,  can  the  user  see  possible  corrections? 

4.  Does  the  portal  support  Back  and  Forward  commands? 

5.  Does  the  portal  support  Auto  Infill  and  Auto  Backfill? 

6.  Does  the  portal  offer  a  phone  number  for  off-line  help? 

7.  Does  the  portal  offer  email  contact  for  on-line  help? 

8.  Does  the  portal  write  error  messages  in  natural  language? 

9.  Can  the  user  see  the  help  screen  and  the  process  screen  at  the  same  time? 

10.  Does  the  interface  have  adequate  contrast  between  background  &  foreground? 
Recognition 

1 .  Does  the  portal  provide  “mouse  overs”  describing  icon  buttons 

2.  Does  the  portal  have  a  search  engine? 

3.  If  there  is  a  search  engine,  does  it  support  advanced  search  capabilities? 

Aesthetic  and  Minimalist  Design 

1 .  Is  the  interface  pleasing  to  the  eye? 

2.  Is  the  interface  uncluttered? 
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Provide  Contact  Information 

1 .  Does  the  interface  provide  personnel  addresses  for  email  contact? 

2.  Does  the  interface  provide  personnel  numbers  for  phone  contact? 

3.  Does  the  interface  provide  fax  numbers? 

Avoid  Information  Overload 

Does  the  interface  “chunk”  information  into  seven  plus  or  minus  2  chunks? 

Provide  the  “Inverted  Pyramid”  Style  of  Writing 

1 .  Does  the  interface  provide  the  most  important  information  at  the  top? 

2.  Does  the  interface  provide  the  least  important  information  at  the  button? 

Keep  Download  and  Response  Time  Low 

Does  information  download  occur  within  five  seconds? 

Make  Content  Easy  to  Visually  Scan 

Are  key  words  and  phrases  easily  visible  (for  example,  by  highlighting)? 

Make  Content  Easy  to  Print 

1 .  Are  pages  easily  marked  so  users  can  print  a  few,  or  a  specified  number  of,  pages? 

2.  Are  printed  pages  free  of  right-sided  or  left-sided  “cut  offs”? 

3.  Are  color  pages  readable  in  black  and  white? 

Dr.  Patricia  Chalmers  developed  this  guidelines  from  work  compiled  by: 

Nielsen,  Jacob  and  Mack,  Robert  L.  1994.  Usability  Inspection  Methods.  New  York, 

NY :  John  Wiley  and  Sons,  Inc. 

Pearrow,  Mark,  2000.  Web  Site  Usability  Handbook.  Rockland,  MA:  Charles  River 
Media,  Inc. 

Shneiderman,  Ben,  1987.  Designing  the  User  Interface.  Reading,  MA:  Addison-Wesley. 


USABILITY  EVALUATION  REPORT 


CONCLUSIONS 

Visibility  of  System  Status.  This  portal  scores  low  in  visibility  of  system  status.  The  portal 
does  not  enable  users  to  know  where  they  are  within  the  portal.  Users  should  be  able  to  know 
where  they  are  at  all  times.  They  should  be  able  to  easily  return  to  previous  topics  without 
having  to  repeatedly  activate  the  “Back”  button.  However,  on  a  few  pages,  the  user  could  link 
back  to  certain  sections  of  the  portal  and  this  is  commendable. 

Consistency.  The  portal  scores  excellent  in  providing  consistency.  Consist  terminology,  color, 
and  layout  are  present  throughout  all  pages.  Only  a  few,  font  changes  were  evident  -  found  in 
the  sections  titled, _ and _ . 


Speaking  the  User’s  Language.  The  portal  scores  low  on  speaking  the  user’s  language  as 
almost  every  page  contained  at  least  one  acronym.  Also,  when  the  evaluator  deliberately  made 
an  error,  users  could  not  understand  the  error  message  because  the  portal  used  system  terms  and 
no  glossary  of  terms  was  available. 

Giving  the  User  Control  and  Freedom.  The  portal  scores  average  on  giving  the  user  control 
and  freedom.  The  interface  provided  some  shortcuts  for  frequent  users. 
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Error  Prevention  and  Error  Recovery.  The  portal  scores  low  on  error  prevention  and  error 
recovery.  The  portal  does  not  offer  a  “Help”  section,  on-line  support,  or  off-line  support.  When 
filling  out  forms,  many  actions  were  irreversible.  Users  could  not  be  save  forms.  Auto  infill  and 
backfill  features  were  not  available  for  forms.  Error  messages  were  confusing  and,  in  many 
instances,  the  portal  did  not  offer  steps  to  recover  from  errors.  The  forms  contained  no 
instructions  for  infill.  However,  Back  and  Forward  commands  were  available  and  this  is 
commendable. 

Recognition.  The  portal  scores  low  in  recognition  rather  than  recall.  The  portal  has  no  links  to 
describe  icons  in  case  the  user  forgets  the  meanings  of  icons.  In  addition,  the  portal  supports  no 
search  engines  and  advanced  search  capabilities. 

Aesthetic  and  Minimalist  Design.  The  portal  scores  above  average  on  aesthetic  and  minimalist 
design.  Some  pages  have  light  blue  background  with  dark  blue  text.  This  may  cause  difficult  or 
slow  reading  of  the  text.  However,  there  are  only  a  few  pages  with  this  problem.  The  home 
page  looks  somewhat  disjointed  and  cluttered,  but  most  pages  are  easy  to  view  and  present  no 
readability  problems. 

Provide  Contact  Information.  The  portal  scores  poor  on  providing  contact  information.  Some 
pages  have  telephone  numbers  and  some  pages  have  email  addresses.  However,  the  portal  scores 
high  on  providing  Portal  master  email  contact  information,  as  this  information  appears  on  every 
page.  However,  when  I  tried  to  email  the  Portal  master,  no  one  replied  for  three  days. 

Keep  Download  Response  Time  Low.  The  portal  scores  excellent  on  keeping  download  and 
response  times  low.  All  information  downloaded  within  three  seconds. 

Make  Content  Easy  to  Visually  Scan.  The  portal  scores  average  on  making  content  easy  to 
visually  scan.  Occasionally  I  needed  to  decrease  the  size  of  the  side  bar  or  scroll  to  the  right  in 
order  to  see  complete  pages.  However,  the  use  of  color  changes,  font  size  changes,  and  bolding 
do  make  important  information  visible.  We  commend  the  portal  for  being  easy  to  visually  scan. 

Make  Content  Easy  to  Print.  I  tried  to  print  a  few  select  pages,  guessing  at  the  page  numbers, 
as  they  were  not  available.  I  misjudged  and  printed  the  wrong  pages.  Some  pages  did  not  offer 
the  “printer  friendly”  option  and  came  off  the  printer  with  information  cut  off  on  the  right  side. 

RECOMMENDATIONS 

Know  Your  Users.  It  is  essential  to  know  your  users.  The  first  question  to  answer  is,  “Who  do 
we  want  our  users  to  be?  The  portal  development  team  should  identify  desired  users,  canvas  a 
sampling  of  those  users,  and  request  their  feedback  via  a  quick  and  anonymous  form.  You  may 
not  be  able  to  design  your  portal  for  everyone  in  your  organization,  but  you  do  want  to  design  it 
for  as  many  people  as  possible,  especially  for  users  who  you  are  targeting.  If  the  portal  could 
make  the  most  important  group  of  users  happy,  who  would  those  users  they  be?  The  sooner  you 
identify  your  key  users,  the  more  time  and  cost  you  will  save. 


18 


Know  What  Your  Users  Need  to  Do  to  Accomplish  Their  Work.  Equally  important  is 

knowing  what  your  users  need  to  do  to  accomplish  their  tasks.  The  next  question  to  answer  is, 
“What  is  the  purpose  of  the  portal?  For  example,  is  the  purpose  to  convince  users  to  do 
something?  Or  to  provide  information?  Or  to  seek  input?  All  of  the  above?  Answering  these 
questions  and  providing  fast  and  easy  ways  for  your  users  to  accomplish  their  tasks  will  help 
your  portal  be  most  effective. 

Visibility  of  System  Status.  Users  should  know  where  they  are  in  a  process  at  all  times.  You 
can  help  users  in  a  number  of  ways.  For  example,  the  use  of  page  identifiers  (for  example,  by 
tabs  which  show  all  levels  the  user  has  gone  down)  is  one  way.  Use  of  breadcrumbs  is  another 
way.  Also,  noting  that  the  portal  is  processing  a  request  and  the  time  it  expects  to  take  is  another 
way  to  provide  the  user  with  system  status.  The  use  of  a  slide  bar  is  yet  another  way. 

Consistency.  There  is  excellent  consistency  of  terminology,  color,  and  layout.  The  evaluator 
recommends  these  consistencies  remain.  The  one  exception  is  that  layout  at  various  levels  vary 
slightly  from  level  to  level.  However,  these  inconsistencies  can  enhance  understanding  of 
differences  in  the  levels  if  they  are  not  too  different  from  the  portal’s  main  look  and  feel.  There 
are  a  few  instances  of  font  changes.  The  evaluator  recommends  these  sections  comply  with  the 
consistent  font  of  the  other  pages. 

Speaking  the  User’s  Language.  Knowing  that  at  least  25%  of  the  hits  for  this  portal  come  from 
new  employees,  speaking  the  user’s  language  will  be  extremely  important.  Therefore,  avoid  all 
acronyms,  or  at  least  provide  a  mouse  over  for  acronyms  if  your  stakeholders  insist  on  using 
acronyms  .  This  will  enable  all  users  to  quickly  and  easily  comprehend  portal  contents. 

Error  Prevention  and  Error  Recovery.  The  evaluator  recommends  the  portal  maintain  the 
“Back”  and  “Forward”  buttons  as  these  are  helpful  to  many  users,  especially  if  they  are 
navigating  a  new  part  of  the  portal.  The  evaluator  also  recommends  the  portal  incorporate  a 
“Help”  section,  on-line  support,  and  off-line  support  for  error  prevention  and  recovery.  Also, 
when  an  error  occurs  the  system  should  inform  users  of  1)  the  error,  2)  how  the  error  occurred, 
and  3)  how  the  user  can  recover  from  the  error. 

Form  fill-in  is  especially  prone  to  errors,  especially  if  users  are  filling  out  a  form  for  the 
first  time.  A  form  that  does  not  promote  easy  error  recovery  can  frustrate  users  quickly. 
Therefore,  the  evaluator  recommends  the  portal  provide  instructions  (e.g.,  “no  dashes”  for  a 
phone  number  entry)  and  examples  (e.g.,  “dd/mm/yyyy”).  Allow  users  to  save  their  filled  in 
forms.  Also,  provide  auto  infill  and  backfill  features. 

Recognition.  The  evaluator  recommends  the  portal  provide  a  glossary  in  case  the  user  cannot 
recognize  or  remember  the  definition  of  a  term.  The  evaluator  recommends  a  glossary  via  a 
“mouse  over”  so  users  do  not  have  to  lose  their  train  of  thought,  which  occurs  when  users  need 
to  leave  a  page,  go  to  another  page  to  find  a  definition,  then  return  to  their  original  page,  and 
incorporate  the  definition  into  the  sentence.  The  evaluator  recommends  a  search  engine  with 
advanced  search  capabilities  to  support  users  who  may  remember  only  parts  of  a  keyword  or 
keyword  phrase.  Since  no  provision  for  legends  describing  icon  buttons  are  present,  the 
evaluator  recommends  mouseovers  describing  icon  buttons. 
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Aesthetic  and  Minimalist  Design.  The  portal  scores  above  average  on  aesthetic  and  minimalist 
design,  providing  adequate  contrast  between  background  and  foreground  on  all  pages.  Pages 
were  clear,  simple,  and  users  could  easily  and  efficiently  read  all  text.  There  were  very  few 
cluttered  pages. 

Provide  Contact  Information.  The  evaluator  recommends  the  system  give  users  an  email 
addresses,  phone  numbers,  and  a  fax  numbers  to  enable  users  to  reach  the  appropriate  points  of 
contact.  View  this  recommendation  in  light  of  the  answers  to  the  questions,  “What  is  the 
purpose  of  the  portal?”  If  the  answer  is:  to  “sell  a  product  or  service,”  to  “seek  user  input,”  to 
“provide  a  meeting  place,”  or  to  “draw  in  users”  then  I  would  recommend  complete  contact 
information  on  most,  if  not  all,  pages. 

Keep  Download  Response  Time  Low.  On  this  guideline  the  portal  scores  excellent.  All 
information  downloaded  within  three  seconds.  This  is  highly  commended. 

Make  Content  Easy  to  Visually  Scan.  The  evaluator  recommends  the  portal  reduce  the  size  of 
sidebars  or  eliminate  sidebars  completely.  In  general,  other  techniques  for  making  information 
easy  to  scan  include:  highlighting,  shading,  bolding  italicizing,  or  underlining. 

Make  Content  Easy  to  Print.  The  evaluator  recommends  enabling  users  to  print  a  few  select 
pages,  without  having  to  guess  at  page  numbers.  In  addition,  a  “printer  friendly”  option  should 
be  available. 


Iterative  Evaluations:  The  evaluator  recommends  two  more  usability  evaluations:  once  after 
the  first  prototype  is  developed  (interim  evaluation)  and  once  again  before  sending  out  the 
“finished  product”  (summative  evaluation). 


Usability  Testing.  The  evaluator  also  recommends  usability  testing  as  part  of  the  evaluation. 
Usability  testing  enables  the  portal  development  team  to  measure  improvement  in  user 
performance.  For  example,  testing  can  include  counting  the  number  of  keystrokes  needed  for 
users  to  find  what  they  want  in  a  sample  scenario,  then  comparing  the  numbers  at  different 
phases  of  development.  These  comparison  numbers  will  give  the  development  team  feedback 
regarding  the  effects  of  their  changes.  The  evaluator  also  recommends  cognitive  walk-throughs. 
These  involve  users  telling  the  evaluator  what  they  are  thinking  when  they  try  to  complete  a  task, 
find  a  page,  or  try  to  find  information  on  a  page.  The  evaluator  also  recommends  that 
test/evaluation  measures  be  quantifiable. 


THANK  YOU!  I  would  like  to  again  thank  you  for  the  opportunity  to  evaluate  your  portal.  I 


found  your  development  team,  senior  leaders,  and  end  users  very  dedicated  and  a  pleasure  to 
work  with.  I  look  forward  to  working  with  you  again  at  the  next  stage  of  your  development. 
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APPENDIX  B 
SAMPLE  TABLES 

Note  the  table  in  Figure  1  below.  This  fictitious  table  shows  is  impressive;  however,  it  may  be 
hard  for  some  people  to  picture  the  results. 

Figure  1.  Tables  Showing  Results 


Category 

Score  Before  Redesign 

Score  After  Redesign 

Satisfaction 

1.0  on  a  Score  of  1-7 

2.0  on  a  Score  of  1-7 

Excitement 

1.0  on  a  Score  of  1-7 

7.0  on  a  Score  of  1-7 

Simulated  Task  Accomplishment 

10%  Able  to  Complete 

60%  Able  to  Complete 

Desire  to  Buy 

40%  Want  to  Buy 

70%  Want  to  Buy 

Look  at  Figure  2  below.  This  is  the  same  table  with  red,  yellow,  and  green  colors  to  indicate, 
alarm,  caution,  or  commendation. 


Figure  2.  Color-coded  table  SI 

(lowing  Results 

Category 

Score  Before 
Redesign 

Score  After 
Redesign 

STATUS 

AFTER 

REDESIGN 

Satisfaction 

1.0  on  a  Score  of 

1-7 

2.0  on  a  Score  of  1-7 

Excitement 

1.0  on  a  Score  of 

1-7 

7.0  on  a  Score  of  1-7 

Simulated  Task 
Accomplishment 

10%  Able  to 
Complete 

60%  Able  to 

Complete 

Desire  to  Buy 

40%  Want  to  Buy 

70%  Want  to  Buy 

Remember,  however,  that  many  people  are  color  blind  and  the  most  common  form  of  color 
blindness  is  an  inability  to  differentiate  red  from  green.  Also,  colors  mean  different  things  in 
different  cultures,  so  know  your  audience. 

A  better  chart  might  include  icons,  such  as  a  smiling  face,  frowning  face,  and  a  straight  line 
across  the  mouth  for  something  in  between.  See  the  next  page  for  an  example  of  a  chart. 

See  the  next  page  for  an  example  of  a  chart,  vice  a  table. 
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APPENDIX  B 
SAMPLE  BAR  CHART 

Bar  charts,  rather  than  tables,  may  be  easier  for  most  people  to  understand  at  a  glance.  Note  the 
bar  chart  below  in  Figure  3.  This  fictitious  chart  shows  changes  in  the  user’s  desired  qualities 
before  the  user’s  portal  was  redesigned  (light  blue  bars),  and  after  the  user’s  portal  was 
redesigned  (dark  blue  bars).  In  general,  bar  charts  are  easier  for  people  to  picture.  See  the  next 
page  for  another  idea. 


Figure  3.  Bar  Chart  Showing  Results 
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QUESTIONS  AND  ANSWERS 

Feel  free  to  ask  questions  during  this  course.  However,  if  you  think  of  additional  questions  after 
each  module,  I  will  address  them  at  the  end  of  the  session  as  time  permits. 

Activity:  Please  write  your  questions  below 

Discussion:  The  presenter  answers  final  questions  from  the  audience 
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FEEDBACK 

Estimated  time  to  complete  this  survey:  3  minutes. 

Please  answer  the  questions  below,  then  tear  the  page  out  of  this  book  and  turn  it  in  to  the 
Feedback  Box.  I  very  much  appreciate  your  feedback  so  I  can  improve  my  presentation! 


Number  of  years’  experience  in  usability _ 

Modules  that  presenter  could  cover  more  quickly  (circle  any  that  apply) 

1 -Introduction  2-Definitions  3-App lying  Tutorial  4-Motivations  for  Metrics 

5- Determine  Your  Metrics  6-Determine  Your  Method  7-Implement  Method 

8-Evaluating  Your  Metrics  9-Evaluating  Your  Methods  10-Reporting  Findings 


Modules  that  presenter  could  cover  more  slowly  (circle  any  that  apply) 

1 -Introduction  2-Definitions  3-Applying  Tutorial  4-Motivations  for  Metrics 

5- Determine  Your  Metrics  6-Determine  Your  Method  7-Implement  Method 


8-Evaluating  Your  Metrics  9-Evaluating  Your  Methods  10-Reporting  Findings 


Your  satisfaction  with  the  content  of  the  material  presented  (circle  one) 

extremely 

very 

somewhat  Neutral 

somewhat 

very 

extremely 

unsatisfied 

unsatisfied 

unsatisfied 

satisfied 

satisfied 

satisfied 

Your  satisfaction  with  the  timing  of  the  material  presented  (circle  one) 

extremely 

very 

somewhat  Neutral 

somewhat 

very 

extremely 

unsatisfied 

unsatisfied 

unsatisfied 

satisfied 

satisfied 

satisfied 

Your  physical  comfort  during  the  presentation  (circle  one): 

I  was  uncomfortable  0%  of  the  time  25%  50%  75%  100%  of  the  time 

Reason: 


Your  level  of  energy  during  the  class  (circle  one) 

extremely  tired  very  tired  somewhat  tired  somewhat  alert  very  alert  extremely  alert 


What  can  the  presenter  do  to  enhance  this  tutorial? 


What  can  the  presenter  eliminate  from  this  tutorial? 


Was  there  anything  you  liked  about  the  tutorial?  If  so,  what  did  you  like? 


Are  there  any  further  comments  you  would  like  to  add?  Use  back  of  this  page  if  needed 


MANY  THANKS  FOR  YOUR  TIME  IN  COMPLETING  THIS  FEEDBACK  AND 
GOOD  LUCK  IN  YOUR  USABILITY  PROJECT/S! 
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Software  Reliability  Definitions 
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■  The  American  Institute  of  Aeronautics  and  Astronautics  (AIAA)  - 
“the  application  of  statistical  techniques  to  data  collected  during 
system  development  and  operation  to  specify,  predict,  estimate, 
and  assess  the  reliability  of  software-based  systems.” 

-  IEEE  1633: 

-  (A)  The  probability  that  software  will  not  cause  the  failure  of  a 
system  for  a  specified  time  under  specified  conditions. 

-  (B)  The  ability  of  a  program  to  perform  a  required  function  under  stated 
conditions  for  a  stated  period  of  time. 

-  I  EC  62628: 

-  Software  Dependability  -  ability  of  the  software  to  perform  as  and  when 
required  when  integrated  in  system  operation 


NOTE:  Software  Dependability  includes  Software  Reliability  as  well  as 
other  measures  of  software  performance  and  capability 
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Reliability  Growth  Introduction 
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■  Purpose  of  Reliability  Growth  Planning  is  to  develop  a  reliability 
growth  planning  curve  which  specifies  the  plan  for  achieving 
specified  reliability  values 

■  Provides  a  means  for  tracking  reliability  growth  and  monitoring 
progress  as  the  test  proceeds 

■  Growth  is  achieved  from  software  design  changes  that  effectively 
correct  software  defects  that  cause  system  failures 

■  Detailed  guidelines  are  provided  in  MIL-HDBK-189 

■  Example  of  a  reliability  growth  planning  curve  is  provided  in  a 
figure  (modification  of  MIL-STD-781,  Figure  103-1)  on  the  next 
slide. 


Reference  MIL-STD-781  D,  Task  103 
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Traditional  Reliability  Growth  Curve 


Modified  Graph  from  MIL-STD-781 


Page  5 


MTBF  (Hours) 


Notional  Software  Reliability  Growth  Raytheon 

Curve  Divided  into  Phases 
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Reliability  Assessment 
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Integration  of  the  Software  Reliability  Growth 
into  Software/System  Development  Process 


Software  Development  Process 


#o 

o 

Process  Characteristics  CMM 

Q) 

Level  & 

O 

U 

KSLOC  Estimates 

S 

a 

Initial  IOS  Estimates 

o 

Software  Development  Defects 
Data  Collection 

Updated  10! 

5  Estimates 

Test  Execution  Time  &  Time 
Until 

Failure  Data  Collection 

IOS  Actual  Measurements 

Capability  Maturity  Model 
Step  1 


Rayleigh  Mode 

Ste 

‘I/SWEEP  Tool 

P  2 

r 

CASRE  Tool  Set  Models 
Step  3 


Software  Reliability 

Estimation/Performance 

Evaluation 

Software  Reliability 

Estimation/Performance 

Evaluation 

Software  Reliability 

Estimation/Performance 

Evaluation 

1 

r 

r 

Reliability  Models, 
Availability  Assessment, 
Requirements  Validation 


Reliability  Models, 

Availability  Assessment, 

Requirements  Validation 

Reliability  Models, 
Availability  Assessment, 
Requirements  Validation 
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IEEE  1633 
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IEEE  1633  -  IEEE  Recommended  Practice  on  Software 
Reliability  (SR) 

□  Developed  by  the  IEEE  Reliability  Society  in  2008 

□  Purpose  of  IEEE  1633 

>  Promotes  a  systems  approach  to  SR  predictions 

>  Although  there  are  some  distinctive  characteristics  of  aerospace 
software,  the  principles  of  reliability  are  generic,  and  the  results  can 
be  beneficial  to  practitioners  in  any  industry. 


IEEE  1633  is  Useful  for  Tailoring  Software  Reliability  Prediction  & 
Measurement  Processes  Depending  on  the  Particular  Needs 
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How  to  Apply  IEEE  1633  to  Align  with  Your  SW 
Development  Process 
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3  step  process  leveraging  IEEE  1633: 


>  Step  1  -  Keene  Model  [for  early  software  predictions 


•  Weighs  SEI  CMMI  Process  Capability  (e.g.  CMMI  Level  5 
achieved  by  IDS)  to  Software  Size  (e.g.  lOKSLOCs) 

>  Step  2^  SWEEP  Tool  Ifor  tracking  growth  of  Software  Trouble 


Reports  (STRs)  and  Design  Change  Orders 


>  Step  3  -  CASRE  Toollfor  tracking  failures  in  software  testing 


Software  Reliability  Prediction  &  Measurement  Process 
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Capability  Maturity  Model  (Keene  Model) 
Step  1 
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■  The  Capability  Maturity  Model  provides  a  preliminary  prediction 
based  on: 

-  Estimated  size  of  the  code  in  KSLOC 

-  Software  Engineering  Institute’s  (SEI)  Capability  Maturity  Model 
(CMM)  rating  of  the  software  developer 

-  The  assertion  is  that  the  software  process  capability  is  a  predictor  of 
the  latent  faults  shipped  with  the  code. 


Defect  Rate 


SEI  Level  V 


The  higher  the  SEI  Level  the  more  efficient  and 
Organization  is  in  detecting  defects  early  in  development 


The  better  the  process,  the  better 
the  process  capability  ratings  and 
the  better  the  delivered  code  will 
perform . . .  .defects  will  be  lower. 


Time 


Keene  Model  is  Useful  for  Establishing  Software  Reliability  Requirements 
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SWEEP  (Software  Error  Estimation  Program)  Rav"»eon 
Step  2 


■  The  SWEEP  tool  enables  you  to: 

-  Predict  and  track  the  rate  at  which  defects  will  be  found 

-  Predict  the  latent  defect  content  of  software  products 

-  Plot  design  changes  over  time  -  demonstrate  growth 

-  Analyze  estimated  errors  injected  in  each  phase  of  the  software 
development  cycle 

-  Determine  the  detection  effectiveness  and  leakage  of  errors  to 
subsequent  phases. 

-  Measure  percentage  of  critical  failures  that  feedback  into  the 
Keene  model 

■  Data  Collection 

-  Data  is  typically  collected  using  Software  Trouble  Reports  (STR) 

-  Data  can  be  organized  by  development  phase  or  time  increments. 


SWEEP  is  Useful  for  Curve  Fitting  Software  Coding  and  Test 
Defect  Data  With  the  Rayleigh  Distributions 
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CASRE  (Computer  Aided  Software  Reliability  Raytheon 

Estimation)  -  Step  3 

■  CASRE  is  a  software  reliability  measurement  tool  which  collects  and 
analyzes  software  test  failures  over  time  or  phases  in  development 

■  The  modeling  and  analysis  capabilities  provided  by  the  reliability  package 
SMERFS  (Statistical  Modeling  and  Estimation  of  Reliability  Functions  for 
SW). 

■  In  implementing  CASRE,  the  original  SMERFS  user  interface  has  been 
discarded,  and  the  SMERFS  modeling  libraries  have  been  linked  into  the 
user  interface  developed  for  CASRE. 

■  Data  Collection  -  CASRE  can  accept  two  types  of  data  files: 

-  Times  between  successive  failures 

-  Failure  counts  per  test  interval 

■  Experience  shows  that  at  the  start  of  software  test,  modules  having  more 
than  about  2000  source  lines  of  executable  code  will  tend  to  have  enough 
faults  to  produce  at  least  40  to  50  failures. 


CASRE  is  Useful  for  Curve  Fitting  Software  Test  Failure  Data 

with  Multiple  Distributions 
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Sample  CASRE  Data  Models  and  Output 


CASRE  Curve  Fitting 
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Software  Errors  Remaining  / 100  KSLOC 
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Correlation  Between  the  3  Steps 
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Actual  data  supports  the 
Rayleigh  Model 


Keene  Development 
Process  Software 
Prediction  Model  Results 
Correlation  from  Curt 
Smith  paper  at  ISSRE  99 


▲  SWEEP  actuals 

—  SWEEP  prediction 
•  CASRE  actuals 

—  Generalized  Poisson  estimate 

—  DPM  prediction 


DPM  =  Development 
Process  Model 
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Figure  reproduced  by  permission  of  the  author  and  the  IEEE  ©  10th  International  Symposium  on  Software  Reliability  Engineering 
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Sample  Results  that  Demonstrate  Growth 
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Demonstrates  Traditional  Growth  Curve 
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IOS  and  Ao  Calculations 
Added  Steps  4  and  5 


Raytheon 


Closed  Loop  System  with  Step  5  Feedback  to  Steps  1  -3 
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Value  Added  Benefits 


Raytheon 


■  Reduce  cost  of  failures  later  in  the  software 
development  process 

■  Track  failure  trends  during  all  phases  of  software  test 
focusing  on  probabilistic  conditions  (e.g.  race  conditions) 
and  systemic  process-related  issues 


■  Drive  software  design  corrective  actions  to  improve 
reliability  results  in  a  lower  customer  Total  Cost  of 
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Software  Reliability  Innovation  -  Path  Forward 

Initiatives  to  Increase  SW  Reliability  Growth  and 
Accelerate  Deliveries  of  Mature  /  Dependable  SW  to  the 
Warfighter: 

□  Continue  improvement  of  software  reliability  growth  testing 
processes  and  tools 

□  Continue  decreasing  SW  fault  density  significantly  during  SW 
production,  prior  to  testing 

□  Continue  to  develop  new  standards  or  sustain  existing 
standards  (e.g.  IEEE  1633  and  IEC  62628) 

□  Develop  more  rigorous  software  development  processes 


Raytheon  Approach  Accommodates  Increased  SW  Complexity  &  Reliability 
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SW  Reliability  References 
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•  Metrics  and  Models  in  Software  Quality  Engineering,  Stephen  Kan, 
Addison  Wesley  Publishing 

•  Handbook  of  Software  Reliability  Engineering,  Michael  Lyu, 
McGraw  Hill  Publishing 

•  Software  Reliability:  Measurement,  Prediction,  Application,  John  D. 
Musa,  Anthony  lannino,  and  Kazuhira  Okumoto,  McGraw-Hill  Book 
Company 

•  IEEE  1633:  Recommended  Practice  on  Software  Reliability  (SR) 

•  IEC  62628:  Guidance  on  Software  Aspects  of  Dependability 


Page  1 9 


Raytheon 

Biography 

-  Lou  Gullo,  Raytheon,  Integrated  Defense  Systems,  Whole  Life 
Engineering  Directorate.  Leader  on  several  Enterprise-wide 
Engineering  Council-sponsored  special  projects  including  software 
reliability  methods  and  the  automation  of  electrical  stress  analysis 
methods.  30  years  experience  in  military,  space  and  commercial 
programs.  Retired  US  Army  Lieutenant  Colonel.  Senior  Member  of  the 
IEEE.  IEEE  Reliability  Society  Standards  Committee  Chair.  Member  of 
the  Reliability  and  Maintainability  Symposium  (RAMS)  Management 
Committee. 

Louis  J  Gullo 

Sr  Principal  Systems  Engineer 
Lou.Gullo@Raytheon.com 
401-842-4139 


Page  20 


Untitled  Document 


Files  are  in  Adobe  format 
Download  the  newest  version  from  Adobe . 


27th  Annual  National  Test  and  Evaluation  Conference 

“Test  &  Evaluation:  Serving  the  Warfighter” 

Tampa,  FL 

14  -  17  March  2011 


Agenda 


MONDAY.  MARCH  14.  2011 
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SESSION  B:  OTA’S  (OPERATIONAL  TEST  AGENCY’S)  ROUNDTABLE 

Session  B  Chair  and  Roundtable  Moderator:  Dr.  Catherine  Warner.  Science  Advisor,  DOT&E,  OSD 
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11554  -  The  Impact  of  High  Accuracy  Target  Geometry  in  Modeling  and  Simulation  to  Support  Live  Lire  Test  and  Evaluation,  Mr.  Scott 
Homung.  U.S. Army  Research  Laboratory/  SLAD 
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Developmental  Testing 
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Ernest. swauger@jpeocbd.  osd.  mil 
rmim-6440 


Date  of  Briefing: 


17  March  2011 


*  Provide  Information  for  informed  decision  making 

*  Verification  and  Validation  of  the  Systems  Engineering 
Process 

*  Provide  confidence  that  the  system  design  solution  is  on- 
track  to  satisfy  the  desired  capabilities 


Reduce  Technical  Risk 
and  increase  probability 
of  a  successful  program 
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Understanding  the  Capability 


\ 


\ 


•  Use  Developmental  Testing  to: 

—  To  stress  the  system  and  understand  its  limitations 

—  To  operate  it  for  sufficient  periods  of  time~in  a  relevant 
environment--to  derive/predict  future  RAM 

—  To  prosecute  it  in  a  relevant  environment  to  understand  its 
survivability  and  safety  in  combat 

*  To  know  why  it  works/how  it  works 

*  To  ensure  the  delivered  performance  can  withstand 
scrutiny  from  challenges  by  Users/Industry 

•  To  ensure  that  it  can  repeat  its  performance  under 
operational  conditions  with  actual  operators 
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Laboratory  Tests 


*  Controlled  environments 

-  Temp,  Humidity,  Air  Pressure,  Concentrations,  etc. 

*  Measured  inputs,  outputs 

-  Linear  and  non-linear  events  in  linear  and  non  linear 
methodologies  ? 

*  Methodology  (Test  Procedures,  Test  Data,  and  Test  Tools)  needs  to 

be  specific 

-  Include  calibration,  identification  of  all  parts  and  components, 
and  detailed  procedures  for  all  steps 

-  Functional  and  interoperable  requirements 

-  Referenced  Standards 

-  Derived  test  requirements  based  on  the  functional  and 
interoperable  requirements  and  referenced  standards 

-  Assumptions  which  may  influence  the  selection  of  a  specific  test 
method  or  scope  of  testing 
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Demonstrate,  Test,  Evaluate  and 
Introduce  Technology _ 


•  Potential  solution  is  tested/evaluated  to  determine 
how  well  it  addresses  the  intended  functional 
requirement 


*  Introduction  of  solution  into  practice 

*  Develop  performance  standards  and  guides,  as 
appropriate,  to  ensure  safety  and  effectiveness 


Not  all  new  solutions  will  require  the  publication 

of  new  standards  and  guides 
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Performance  vs.  Method  Standards 

\  \sS 


Performance  vs.  Method  Standards 

-Performance:  Limit  to  be  met 

-  Methodology:  Detailed  description  of  how  a 
test  is  to  be  conducted,  under  what 
conditions,  calibration  accuracy  and  interval, 
materials,  etc. 


Development,  validation,  and  application  of  new 
analytical  methods 


•  Development  of  new  analytical  reference  materials 
and  operation  of  proficiency  testing  programs 


Full  range  of  sophisticated  measuring  techniques  and 
state-of-the-art  laboratories 


•  International  Standards  Organizations 

-  ISO,  I  EC,  ITU,  etc. 

-  ASTM  International 

*  Regional  Standards  Organizations 

-  CEN,  ETSI,  IRMM,  PASC,  etc. 

•  National  Standards  Bodies 

-  ANSI,  JISC,  NFPA 

*  Standard  Developing  Organizations/Bodies: 
-IEEE,  DoD 

-Peer  Reviewed,  Inclusive  of  all  Stakeholders 


DoD  Policy:  Use  NGO  STDs 

.  \  \  _ 


DoD  Policy  on  the  Use  of  Non-Government  Standards:  DoD 
is  committed  to  the  adoption  and  use  of  voluntary 
consensus  standards  (defined  in  DoD  4120.24-M  as  "non- 
Government  standards  (NGS)"),  where  practical,  instead 
of  developing  new  or  updating  existing  government 
specifications  and  standards.  This  policy  is  consistent  with 
P.L.  104-113,  the  National  Technology  Transfer  and 
Advancement  Act  of  1995  (NTTAA)  and  with  Office  of 
Management  and  Budget  (OMB)  Circular  No.  A-119 
(Revised),  "Federal  Participation  in  the  Development  and 
Use  of  Voluntary  Consensus  Standards  and  in  Conformity 
Assessment  Activities,"  dated  February  10, 1998. 
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Repeatability,  Reproducibility 


Repeatability:  Ability  to  repeat  test  in  same 
laboratory,  same  tester 


Reproducibility:  Ability  to  reproduce  test  in  different 
laboratories,  different  testers 


Parameters  to  be  included  in  future  test  reporting? 


•**<*/• 


Lab  Management  Practices 


•  How  Do  We  Ensure  Quality  Control  in  Processes? 

—  ISO/IEC  17025:  Industry  standard,  used  also  by  facilities 
across  the  gov't 

*  ISO  17025  is  the  leading  international  laboratory  quality  management 
system  (QMS)  standard.  ISO  17025  is  compatible  with,  but  not 
equivalent  to  ISO  9001.  ISO  17025  connects  the  laboratory  quality 
management  system  to  all  other  laboratory  processes. 

*  It  specifies  the  general  requirements  for  the  competence  to  carry  out 
tests  and/or  calibrations,  including  sampling. 

*  It  covers  testing  and  calibration  performed  using  standard  methods, 
non-standard  methods,  and  laboratory-developed  methods. 

*  It  is  applicable  to  all  organizations  performing  tests  and/or  calibrations. 
These  include,  for  example,  first-,  second-  and  third-party  laboratories, 
and  laboratories  where  testing  and/or  calibration  forms  part  of 
inspection  and  product  certification. 

*  It  is  applicable  to  all  laboratories  regardless  of  the  number  of  personnel 
or  the  extent  of  the  scope  of  testing  and/or  calibration  activities. 
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*  There  are  many  commonalities  with  the  ISO  9000 
standard,  but  ISO/IEC 17025  adds  in  the  concept  of 
competence. 

•  The  contents  of  ISO/IEC  17025  -  The  ISO/IEC  17025 
standard  itself  comprises  five  elements  that  are  Scope, 
Normative  References,  Terms  and  Definitions, 
Management  Requirements  and  Technical  Requirements. 
The  two  main  sections  in  ISO/IEC  17025  are  Management 
Requirements  and  Technical  Requirements.  Management 
requirements  are  primarily  related  to  the  operation  and 
effectiveness  of  the  quality  management  system  within 
the  laboratory.  Technical  requirements  includes  factors 
which  determines  the  correctness  and  reliability  of  the 
tests  and  calibrations  performed  in  laboratory. 
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*  Laboratories  use  ISO/IEC 17025  to  implement  a  quality 
system  aimed  at  improving  their  ability  to  consistently 
produce  valid  results.  It  is  also  the  basis  for  accreditation 
from  an  Accreditation  Body.  Since  the  standard  is  about 
competence,  accreditation  is  simply  formal  recognition  of 
a  demonstration  of  that  competence. 


*  A  prerequisite  for  a  laboratory  to  become  accredited  is  to 
have  a  documented  quality  management  system.  The 
usual  contents  of  the  quality  manual  follow  the  outline  of 
the  ISO/IEC  17025  standard. 
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Need  to  Standardize  Test  Processes, 

Facility  Mgmt 


*  It  enhances  credibility 


•  It  enhances  repeatability,  reproducibility 


*  It  defends  against  legal  challenges 


It  supports  efficient  use  of  resources  and  best  data  quality 


\  I  1 

DoD  Process  for  Establishing 

T&E  Standards  _ 


CBRND  T&E  Executive  establishes  DoD  CBRND  T&E  standards 
through  T&E  Capabilities  and  Methodologies  IPT  (TECMIPT) 

SMEs  in  TECMIPT  commodity  area  sub-groups  provide  rigor  to 
T&E  standards  development  Worldwide  cbrnd  t&e 

Standards  Partners 


TE  CMIPT  Members 


Interagency 
partners  - 
DHS,  EPA,  A 
NIST 


DUSA-TE 


4  Sendees 
DT/OT 
Agencies 


AMSAA 
JPEO-CBD 


'W US  Inter- 

W  agency 

Industry  1 

Inter-  J 

DoD  1 

national  A 

DOT&E 


JSTO 


JRO 


How?  TECMIPT  Process 

s  \  '  ' 


Provides  joint,  cross-community  subject  matter  expertise  and 
rigor  to  establish  T&E  Standards 

•  JPEO-CBD,  JSTO,  JRO,  ATEC  (AEC,  OTC,  DTC,  DPG),  M COTEA, 
OPTEVFOR,  AFOTEC,  ECBC,  DOT&E,  NSWC-Dahlgren,  AMSAA  (serves 
as  TECMIPT  Chair  for  DUSA-TE),  NIST,  DHS,  EPA 

•  Identifies  T&E  capability  gaps  for  DUSA-TE's  POM  submission 

•  June  2009  -  Instructions  to  the  TECMIPT: 
Develop/review/recommend  T&E  standards  documents  for  CBRND 
T&E  Executive  approval 

•  July  2010  -  CBDP  T&E  Standards  Development  Plan  signed  into  policy 
by  CBRND  T&E  Executive 

-  Includes  plans  for  QA  -  obtain  outside  certification  of  all  DoD 
CBDP  test  labs 


TECMIPT  Mechanism  forT&E  Standards 

Development 


*  Seven  Capability  Area  Process  Action  Teams  (CAPATs) 
develop/review/recommend  T&E  standards  documents 
for  CBRND  T&E  Executive  approval 

•  CAPATs: 

-  Chemical  Detection 

-  Biological  Detection 

-  Individual  Protection 

-  Collective  Protection 

-  Decontamination 

-  Radiological/Nuclear  (cross-commodity) 

-  M&S  (cross-commodity) 


Explosion  of  Interest  in 
TECMIPT  CBRND  T&E  Standards  Process! 


NIST 


Reliable,  Repeatable, 
Reproducible  Results 


Enables  Data  Sharing 


Reduces  redundant  testing 

Saves  $ 


•**<*/• 


Principles  ajnd  Key  Actions 


\ 


*  Rigorous  Testing  is  Required  to  Ensure  Vendor  and  System 
Compliance  with  Standards 

*  Standards  and  Conformity  Assessment  Processes  Must  be 
Identified  and  Adopted  Across  all  Agencies  to  Ensure  Full 
Interoperability 

*  Timely  Adoption  and  Use  of  Appropriate  Standards  is 
Critical  to  Achieving  Goals 


Operational  Test  Agency 

Roundtable 


Moderator:  Dr.  Catherine  Warner 
Science  Advisor 

Director  of  Operational  Test  and  Evaluation 


NDIAT&E  Conference  March  15,  2011 
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Roundtable  Participants 


*  Major  General  Genaro  Dellarocco,  USA,  Commander,  ATEC 

*  Rear  Admiral  David  Dunaway,  USN,  Commander,  OPTEVFOR 

*  Major  General  David  Eichhorn,  USAF,  Commander,  AFOTEC 

*  Colonel  David  Reeves,  USMC,  Commander,  MCOTEA 

*  Colonel  Joseph  Puett,  USA,  Commander,  JITC 
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Current  DOT&E  Initiatives 


Testers  Engage 
Early 

Requirements  are: 
Unambiguous 
Testable 

Relevant  to  mission 
accomplishment 
Jechnically  Realisi 


Integrated 

Testing 

Developmental 
Operational 
Live  Fire 


Field  New 
Capabilities 
Rapidly 

Accelerated  Testing 


Improve 

Suitability 

Reliability  Growth 


Today's  focus  -  Integrated  Testing 
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Integrated  Test  and  Evaluation 


•  What  is  it? 

-  Testing  early  in  mission  context  and  realistic  environments 

-  An  efficient  continuum  of  tests  throughout  DT,  OT,  LFT 

-  Using  data  from  one  type  of  test  for  insight  into  other  types 

-  Using  all  test  data  to  support  evaluations 

-  Not  a  replacement  for  independent  OT&E 

*  Why  is  it  important? 

-  Discover  problems  early  when  they  are  cheaper  and  easier  to  fix 

-  Understand  system  performance  across  operational  envelope 

-  Increase  confidence  in  test  results 
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Scientific  Approaches  Necessary  for 
Successful  Integrated  T&E 


*  Structured  and  rigorous  statistical  tools 

-  Stochastic  simulations  to  supplement  field  tests 

-  Methods  for  rigorous  assessments  of  small  sample  sizes 

-  Methods  to  combine  data  from  disparate  sources 

*  Design-of-experiments  (DOE)  principles 

-  Quantitative  response  variables  -  mission-based  for  OT 

-  Breadth  of  coverage  of  the  operational  environment  -  including 
realistic  threats 

-  Methods  for  strategically  varying  operational  conditions 

-  Objective  measures  of  "How  much  testing  is  enough?" 

-  Presentation  of  confidence  based  results 


^tHATlo 


Opening  Question  #1 


*  How  does  your  command  define  the  mission 
context  to  be  used  in  operational  tests? 

-  What  is  your  view  of  how  mission 
accomplishment  should  be  evaluated? 
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Question  #2 


•  How  can  (does?)  your  command  enable  Integrated 
Testing  to  occur  in  realistic  operational 
environments? 

-  How  much  influence  can  (do?)  you  have  on  the 
developmental  test  program? 


A  "Tail"  of  Getting 
Adequate  LFT&E  Funding 


Original  planned  buy 
of  120  C-17, 
approximate 
acquisition  cost  $3B 

-Cost  of  the 
LFT&E  program 

$30M  (1%) 

Eventual  buy  over 
200  aircraft 

-Cost  of  one  tail  of 
one  C-17,  provided 
information  to 
improve  survivability 
for  over  200  aircraft 


^tHATlo 


Question  #3 


*  The  cost  of  DT  and  OT  is  a  small  percentage  of  a 
program's  acquisition  costs;  however  the  cost  of 
testing  is  a  large  percent  of  the  budget  in  the  fiscal 
years  in  which  it  occurs. 

-  The  current  environment  of  efficiencies  appears  to 
exacerbate  concerns  about  the  cost  of  testing. 

•  What  do  you  think  can  be  done  to  increase  the 
relevance  and  perceived  importance  of  government 
testing  both  DT&E  and  OT&E  -  to  demonstrate  its 
"worth"? 
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Question  #4 


•  Interoperability  is  key  to  US  military  operations. 

-  Testing  interoperability  in  a  lab  environment  is 
straightforward.  What  are  your  challenges  with 
testing  interoperability  in  realistic 
environments? 

-  What  can  be  done  in  terms  of  an  incentive 
structure  to  get  PEOs  and  PMs  to  assess  their 
systems  early  on  in  a  joint  interoperability 
laboratory  environment? 
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Question  #5 


*  How  do  you  see  the  role  of  M&S  in  the  conduct  of 
OT&E? 

•  How  can  DT  enable  better  use  of  M&S  tools  in 
OT? 

*  How  do  you  foster  an  appropriate  and  adequate 
VV&A  program/plan? 
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Developmental  Test  &  Evaluation 


Pentagon  Response  to  Strengthening  DT&E 

Presented  to  NDIA  T&E  Conference 

Chris  DiPetto 
Principal  Deputy 

Deputy  Assistant  Secretary  of  Defense,  Developmental  Test  &  Evaluation 

March  15,  2011 
www.acq.osd.mil/dte/ 
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Megatrends 


•  WSARA 

OSD  response 

*  Acquisition  Efficiencies 

Better  Buying  power 


2  Last 


“This  department  simply  cannot  risk  continuing  down  the  same 
path  -  where  our  investment  priorities,  bureaucratic  habits,  and  lax 
attitudes  towards  costs  are  increasingly  divorced  from  the  real 
threats  of  today,  the  growing  perils  of  tomorrow,  and  the  nation’s 

grim  financial  outlook.” 

Secretary  of  Defense  Robert  M.  Gates 


SECDEF  Press  Conference 
6  JAN  2011 
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WSARA  and  DT&E 


Principal  advisor  to  the  Secretary  of  Defense  and  AT&L 

on  DT&E  in  the  DoD 


Responsibilities: 

-  Program  Oversight 

-  Planning  (TEMP  Isl) 

-  Policy  and  Guidance 

-  Acq  DT&E  workforce 

-  Component  Capability 

-  Annual  Report 


DT&E  in  Title  10,  USC,  Section  139d 
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Pentagon  Plan  for  Efficiency 


Target  Affordability  and  Control 
Cost  Growth 

Incentivize  Productivity  & 

Innovation  in  Industry 

Promote  Real  Competition 

Improve  Tradecraft  in  Services 
Acquisition 

Reduce  Non-Productive  Processes 
and  Bureaucracy 


(L)  Secretary  of  Defense  Robert  M.  Gates 
(R)  USD  AT&L  Dr  Ashton  B.  Carter 


“Consumers  are  accustomed  to  getting  more  for  their  money  -  a  more  powerful 
computer,  wider  functionality  in  mobile  phones  -  every  year.  When  it  comes  to  the 
defense  sector,  however,  the  taxpayers  had  to  spend  significantly  more  in  order  to 

get  more.  We  need  to  reverse  this  trend.” 

Secretary  of  Defense  Robert  M.  Gates 
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Implications  for  the  Test  and 
Acquisition  Communities 


Enterprise  will  manage  risk 

-  Rapid  vs.  Deliberate  Acquisition 

Visibility 

-  DT&E  voice  at  DAB 

-  Increased  planning  rigor/fidelity 

-  Efficiencies:  DOE,  IT,  M&S . 

Acquisition 

-  Oversight  and  Accountability 

-  Accept  less  risk  at  MS  decisions 

-  Improving  Process  Effectiveness 

-  More  DT  less  OT? 

-  Confirmation  vs  Discovery 

-  Affordability 
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What’s  new  in  DT&E? 


*  Director,  DT&E  is  now  DASD  (DT&E) 

*  Concurrent  Service  TRMC/DT&E 

*  Major  Leverage  Points 

-  TEMP  Approval 

-  Defense  Acquisition  Board  Engagement 

-  Peer  Reviews 

-  Assessment  of  Operational  Test  Readiness 

*  Annual  Report:  2nd  in  the  hopper 

*  Policy  Changes: 

-  Key  Leadership  Positions 


5  Last 
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DASD(DT&E)  Focus  Areas 


DASD  (DT&E) 

•  T&E  Workforce  (Inherently  Gov  Work) 

•  DAU  T&E  Curriculum 

•  Metrics  -  WSARA  requirement 

•  DOE 


•  Responsible  Test  Organization 

SecDef  Efficiencies 

•  TES  /  TEMP  Consolidation 

•  Use  of  Government  T&E  Facilities 

•  Concurrent  Service  TRMC/DT&E 


10  Last 


DISTRIBUTION  STATEMENT  A  -  Cleared  for  public  release 
by  OSR  on  Jan  14  201 1  -  SR  case  number  1 1-S-0993 


UNCLASSIFIED 


8 


T&E  Workforce: 

Government  Performance  of  Critical  Acquisition  Positions 


T&E  Workforce  divided: 

•  DAWIA  coded 

•  Non-DAWIA  coded 

•  Service  Support  Contractors 


•  aeto«'»“U 

o  pTOp«®U  - 


Mil  Non  T&E 
Coded. 
1% 


Acq  Non  T&E 
(PM,  SPRDE) 
9% 


Military  T&E 
Coded 
4% 


Civilian  Non 
T&E  Coded 
11% 


Civilian  T&E 
Coded 
20% 


Support 

Contractor 

42% 


Data  from  Services’  2009  Self  Assessment 


o  Program  Lead  Test  and  Evaluation 
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DAU  T&E  Curriculum 


T&E  curriculum  out  of  date 


*  DT&E  actively  involved  with  DAU 

*  Training  for  Non-DAWIA  coded  workforce 

*  DAU  T&E  Curriculum  Re-Certified 

*  Incorporate  practical  training 


Improved  training  for  entire  T&E  Workforce 
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Responsible  Test  Organizations 

(RTOs) 

Lack  of  DT&E  expertise  and  impartial  evaluation 

■  Immature  systems  going  into  OT&E 

■  Lack  of  government  access  to  contractor  data 

■  Lack  of  impartial  evaluations  of  system  performance 

■  No  single  government  point  of  contact  for  DT&E 

■  Way  Ahead: 

■  Finalize  RTO  definition  with  T&E  Execs  and  Components 

■  DoD  5000.02  Policy  Change:  Require  designation  of  Gov  RTO 

■  Update  Defense  Acquisition  Guidebook 


Impartial  &  early  reporting  of  system  deficiencies 
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TES/TEMP  Consolidation 


Acquisition  documents  being  streamlining 

■TES  development  duplicates  TEMP  development 

■  Not  enough  transparency  on  T&E  resources  at  Milestone  A 

■  Eliminate  the  TES  at  Milestone  A 

■  Incorporate  into  a  TEMP  at  Milestone  A 


Earlier  planning  &  identification  of  T&E  resources 


DISTRIBUTION  STATEMENT  A  -  Cleared  for  public  release 
by  OSR  on  Jan  14  201 1  -  SR  case  number  1 1-S-0993 


UNCLASSIFIED 


12 


Use  of  Government 
T&E  Capabilities 

Government  paying  twice  for  T&E  capabilities 

■  DoD  owns  a  National  resource  of  T&E  capabilities 

■  $5.6  Billion  investment  and  operating  cost  in  FY10 

■  Need  to  realize  maximum  value  from  capital  investments 

■  Reinforce  and  Require  adherence  to  Policy  &  Guidance 


Improves  capital  utilization  of  existing  facilities 
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DT&E  Opportunities 


RTA’s  Wanted ! 

Rotational  Training  Assignees  sought  for  1  year 
professional  development  assignment  in  OSD  / 

DT&E 


*  1  Yr  minimum  tour  length 

*  DT&E  covers  mission  TDY  costs  WE  WANT  YOU! 

*  Outstanding  Professional  development 

*  Perfect  for  GS-13/14/15  seeking  OSD  experience 

*  Contact  DT&E  for  more  info! 
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DT&E’s  Vision 


•  Improving  Acquisition  Outcomes 

-  “Going  from  red  to  green ...” 

•  Early  &  Continuous  Program  Engagement 


Minimize  Discovery 
In  IOT&E 


Deep 

Insight 


Proven 
Credibility 


inOT 


Early 
Influence 
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Questions? 


OFFICE  OF  THE  SECRETARY  OF  DEFENSE 

OFFICE  OF  THE  UNDER  SECRETARY  OF 
DEFENSE  FOR  ACQUISITION,  TECHNOLOGY 

AND  LOGISTICS 

DEVELOPMENTAL  TEST  &  EVALUATION 


3090  Defense  Pentagon 
Room  5A1076 

Washington,  DC  20301-3090 

Email:  ddre-dte@osd.mil 

www.  acq .  osd .  m  i  I/d  te 


The  right  information,  to  the  right  decision  maker,  at  the  right  time,  for 

better  decisions 
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Back-Up 


DISTRIBUTION  STATEMENT  A  -  Cleared  for  public  release 
by  OSR  on  Jan  14  201 1  -  SR  case  number  1 1-S-0993 


UNCLASSIFIED 


17 


Testing  in  OSD 


Secretary  of  Defense 
The  Honorable  Robert  M.  Gates 


Under  Secretary  of  Defense 
AT&L 


Hon  Ashton  B.  Carter 


Director,  Operational 
Test  and  Evaluation 
Hon  Michael  Gilmore 
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Knowledge  vs  Cost 
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Challenges  to  doing  good? 


1 .  “Testers  like  to  test” 

-  Who  requires,  who  pays? 


2.  “A  dollar  spent  on  test  is  a 
dollar  spent  on  bad  news” 

-  Incentives  matter 


3.  “Testing  is  driving  up 
our  costs” 

-  Now  vs.  later? 


4.  “We  can’t  afford  it  ” 

-  See  #3 
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’  YSt* 


and  how  can  T&E  help? 


•  Enterprise  Perspective 

-  Acquisition  Savings 

-  Mature  Systems 

-  Reliability 

-  Early  discovery 

-  Adequate  testing  (early) 

•  Test  Community  Perspective 

-  Recognize  our  role 

-  Manage  our  appetite 

-  Support  the  risk-based  level  of 
information  needed 

-  Do  our  job  more  efficiently 

•  T&E  Cost 

-  Too  much 

-  Bad  news 

-  Late  T&E  Requirements 

*  T&E  Savings 

-  DOE 

-  Distributed 

-  CRIS 

-  Capital  Utilization 

-  Integrated  Test 
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What's  Wrong  with  Acquisition? 


Cost 


THE  USUAL  (?)  SUSPECTS 


Over  Budget 

-  GAO:  96  MDAPs,  $300B  over  initial  estimates 

Schedule 

Late  to  Need 

-  Getting  capability  to  the  user  to  meet  urgent  needs 

Performance 

Programs  failing  Operational  Test 

-  Suitability  issues 

-  Late  discovery  of  failure  modes 

-  Performance  shortfalls 

-  Interoperability 
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Why  Test? 


Material  Developer 
Decision  Authority  ^ 


^  Warfighter 
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“Testing  is  the  Conscience  of  Acquisition” 
William  J.  Perry  -  former  SecDef 


•  Confirm  Performance 

•  Safety 

•  Capabilities  and  Limitations 


•  Iterate/Mature  the  Design 

•  Failure  Mode  Discovery 


•  Inform  Acquisition  Decisions 


Cyber  Warfare 


Computer  Network  Operations 

-  Months,  days,  hours... uSecs 

-  Attribution 

-  Role?  DoD,  Federal,  Civil 
Attack  (CNA) 

-  Precision  strike 

-  Kinetic  effects 
Defense  (CND) 

-  Cyber  missiles 

-  Mission  critical  tasks,  functions 
Exploitation  (CNE) 

-  Intelligence 


“The  best-laid  defenses  on  military  networks  will  matter  little 
unless  our  civilian  critical  infrastructure  is  also  able  to  withstand 

attacks.” . Deputy  Secretary  Bill  Lynn 
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Cyber  Warfare 


What’s  the  role  for  T&E? 

Scope:  Focus  on  CND  and  MDAPs? 

-  Define  cyber  defense  issues  in  network 
environments 

-  What  systems  are  most  vulnerable? 

-  Weapon  systems? 

-  IT  systems? 

-  Rigorous  cyber  defense  testing 

-  Develop  a  cyber  defense  T&E  framework 

-  Institutionalize  cyber  defense  IT 


With  hundreds  of  legacy  and  new  programs  in  development  each 
entering  our  networks,  we  cannot  afford  the  chaos  of  each  one 
individually  planning  or  just  not  testing  for  cyber  defense. 
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SecDef  Efficiency  Objectives 


•  Deliver  the  warfighting  capability  we  need  for  the  dollars  we  have 

•  Get  better  buying  power  for  warfighter  and  taxpayer 

•  Restore  affordability  to  defense  goods  and  services 

•  Improve  defense  industry  productivity 

•  Remove  government  impediments  to  leanness 

•  Avoid  program  turbulence 

•  Maintain  a  vibrant  and  financially  healthy  defense  industry 
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T&E  Challenges 


*  Rapid  Fielding 

-  Safety 

-  Caps  and  Lims 

*  Emerging  Technologies 

How/where  to  test? 

-  Hypersonics 

-  Autonomous  systems 

-  Weaponized  unmanned  systems 

-  Net-enabled  weapons 

*  Range  Encroachment 

-  OCS  exploration/drilling  ? 

-  Spectrum? 

-  Wind  generators...  !!!!! 


DISTRIBUTION  STATEMENT  A  -  Cleared  for  public  release 
by  OSR  on  Jan  14  201 1  -  SR  case  number  1 1-S-0993 


UNCLASSIFIED 


27 


T&E  Challenges  (continued) 


•  Complex  Systems 

-  System  of  Systems 

-  Interdependent  systems? 

-  Data  fusion 

-  S/W  intensive  systems 


•  Balancing  Adequacy  vs 
Speed  to  Field,  Cost . 

-  DOE? 


Dynamic 

Targeting 

C2ISR 

TTP  Options 
(non-materiel) 


-  How  much  is  enough?  Risk  management 

-  How  much  M&S?  LVC? 

-  Other  tools 
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T&E  Challenges  (continued) 


•  Reliability 

-  50%  of  MDAPs  are  failing  OT  (Suitability) 

-  DOT&E  imperative  -  RAM  growth  testing 

•  Rigor  -  Realistic  Environments? 

-  Stressing  countermeasures  (GPS  jamming), 

clutter . Operationally  relevant  scenarios 

-  Threat  representations 

•  End-to-End  testing 

-  Mission  Context 
-  Mission  threads 

-  Interoperability  and  IA 
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Where  are  We  Going? 


I  R 

P 


Serial  Testing 


Platform-Based 


Threat-Based 


Contract  Compliance 


Interoperability 


Integrated  Test/Training 


►  % 


System  of  Systems 


Complex  Capabilities 


Mission  Context 


Joint  Mission  Thread 


Rapid/Responsiveness 


Our  T&E  process  needs  to  evolve  to  support  faster  product  cycles, 
more  adaptable  products  and  address  challenges 
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DT&E  Challenges/Imperatives 


*  Support  Acquisition  (WSARA) 

-  Robust,  efficient,  risk-based  T&E 

-  Early  engagement 

-  Performance  Assessment  (inform  the  decision  makers) 

*  Support  SecDef  Initiatives  (Efficient  T&E) 

-  Integrated  Test 

-  DOE 

-  Capital  Utilization 

-  M&S,  ground  testing 

-  Distributed  testing 

•  Reliability 

•  Cyber 
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Department  of  Homeland  Security  Testing  and  Evaluation 

Serving  the  Warfighter 


Homeland 

Security 

Science  and  Technology 

March  15,2011 


Mr.  Gary  Carter 
Director,  T&E  and  Standards 
Director,  Operational  T&E 
Science  and  Technology  Directorate 
Department  of  Homeland  Security 

Approved  by  OCC  (CCD) 


'i|§|e\  Homeland 

W  Security  Agenda 

Science  and  Technology 


■  DHS  Warfighters 

■  Department  of  Defense  and  Department  of  Homeland 
Security  Testing  &  Evaluation 

■  Similarities 

■  Differences 


Conclusion 


iPlf  Homeland 
W  Security 

Science  and  Technology 


United  States  Coast  Guard 

IJSCG  safeguards  our  Nation’s  maritime  interests  in  the  heartland,  in  the  ports,  at  sea,  and  around  the 
globe.  They  protect  the  maritime  economy  and  the  environment,  they  defend  our  maritime  borders,  and 

they  save  those  in  peril. 


MH-65  Helicopter 

U.S.  Coast  Guard  photo  by  Petty 
Officer  3rd  Class  Sabrina  Elgammal 


HH-60  Helicopter 

U.S.  Coast  Guard  photo  by 
Petty  Officer  3rd  Class 
David  Weydert 


Response  Boat 

U.S.  Coast  Guard  photo 
by  Petty  Officer  3rd 
Class  Erik  Swanson 
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Homeland 

Security 


Science  and  Technology 


Transportation  Security  Administration 

TSA  protects  the  Nation’s  transportation  systems  to  ensure  freedom  of  movement  for  people 

and  commerce. 


Advanced  Imaging  Technology 

TSA  Photo 
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Homeland 

Security 


Science  and  Technology 


Customs  and  Border  Protection 

CBP  protects  America’s  way  of  life  while  collecting  revenue,  enforcing  intellectual 
property  and  other  laws  at  the  border,  and  facilitating  legitimate  commerce  and  travel. 


Port  of  Long  Beach,  the  largest  and  busiest  port  in  the  US,  is  one 
location  where  OT  is  planned  for  CBP  Advanced  Spectroscopic  Portal 
radiological/nuclear  detection  equipment. 

Photos  Courtesy  CBP 


Homeland 
&ir  Security 


%vds#T' 


Science  and  Technology 


Immigration  and  Customs  Enforcement 

ICE  promotes  homeland  security  and  public  safety  through  the  criminal  and  civil 
enforcement  of  federal  laws  governing  border  control,  customs,  trade,  and  immigration. 


Images  Courtesy  of  ICE 
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Homeland 

Security 

Science  and  Technology 


United  States  Secret  Service 

Dual  mission  “to  safeguard  the  nation's  financial  infrastructure  and  payment  systems  to 
preserve  the  integrity  of  the  economy,  and  to  protect  national  leaders,  visiting  heads  of 
state  and  government,  designated  sites  and  National  Special  Security  Events.” 


Photos  courtesy  US  Secret  Service 
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gH  Homeland  U.S.  Citizens  and  Immigration  Services 

Security  USCIS  provides  accurate  and  useful  information,  grants  immigration  and  citizenship 

- benefits,  promotes  an  awareness  and  understanding  of  citizenship,  and  ensures  the  integrity 

Science  and  Technology  r  •  .  •  , 

^ of  our  immigration  system. 


LIST  OR  MANIFEST  OF  ALIEN  PASSENGERS  FOR  THE  UNITED  STATES 


“SC/S, 

°00'000., 


OMBNo.  1615-OOOS:  Expires  06/30/20 1 1 

G-325,  Biographic  Information 
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C  ltizeiislup/N at  1  o  na  lity 

File  Number 

vwv) 

OMB  No.  16 1 5-0037;  Expires  01/31/12 

1-730,  Refugee/ Asylee  Relative  Petition 


USCIS  OFFICE  ONXY 


OMB  1615-009 6:  Expires  04  30/2012 

G-1041A,  Genealogy 
Records  Request 


Photo  courtesy  USCIS 


Homeland 

Security 

Science  and  Technology 


Federal  Emergency  Management  Agency 

FEMA  supports  the  citizens  and  first  responders  to  ensure  that  as  a  nation  we  work  together  to 
build,  sustain,  and  improve  our  capability  to  prepare  for,  protect  against,  respond  to,  recover 

from,  and  mitigate  all  hazards. 


Natural  Disaster  requiring  intensive  FEMA 
planning  and  coordination  aided  by  integrated 
IT  systems 

FEMA  photo  by  Bob  McMillan 


FEMA  photo  by  Aaron  Skolnik 


FEMA  photo  by  Bill  Koplitz 
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DHS  and  DoD 


Science  and  Technology  Test  &  Evaluation  Similarities 

■  Acquisition  Policy 

■  DoD  Directive  5000 

■  DHS  Directive  102 

■  Acquisition  Framework 

■  Both  Phase/Gate  Processes  with  Acquisition  Decisions 

■  Acquisition  Categories 

■  DoD  based  on  RDT&E  Cost 

■  DHS  based  on  Lifecycle  Cost 

■  Acquisition  Certification 

■  DoD  Defense  Acquisition  University 

■  DHS  developing  Certification  Processes  and  Courses 

■  Require  Operational  Test  and  Evaluation 

■  DoD  by  law  -  Title  X 

■  DHS  by  policy  -  Directive  026  Test  and  Evaluation 


■  Have  a  DOT&E 

■  DoD  by  law  -  Title  X 

■  DHS  by  delegation  -  Delegation  10003 


10 


Homeland 
&ir  Security 


Science  and  Technology 


DHS  and  DoD 

Test  &  Evaluation  Differences 


J LL 


H.  R.  5005 

One  Hundred  3eoenth  Congress 
of  the 

United  States  of  America 

AT  THE  SECOND  SESSION 


Sec.  302(12)  S&T 
responsible  for 
“...coordinating  and 
integrating  all  research, 
development, 
demonstration,  testing, 
and  evaluation  activities 
of  the  Department ...” 


DHS  T&E  Law 


iPlf  Homeland 
W  Security 

Science  and  Technology 


DHS  and  DoD 

Test  &  Evaluation  Differences 


No  DHS  Operational  Test  Agency 


■  Component  internal  test  organization 

■  DoD  OTAs 

■  Non-system  contractors 
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iPlf  Homeland 
W  Security 

Science  and  Technology 


DHS  and  DoD 

Test  &  Evaluation  Differences 


Systems  Under  Test  often  Interface 
Directly  with  the  Public 


•  Systems  must  not  only  detect,  but  cannot  hold  up 
commerce 

■  Cannot  invade  “privacy” 

■  Public  opinions  often  in  the  news 
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iPlf  Homeland 
W  Security 

Science  and  Technology 


DHS  and  DoD 

Test  &  Evaluation  Differences 


Operational  Testing  Conducted  in  Actual 

Environment 


■  Must  continue  to  do  mission 

■  Cannot  interfere  with  daily  operations 

■  Test  may  encounter  actual  threat 

■  Explosives 

■  Undocumented  aliens 

■  Tests  very  realistic  but  Tester  loses  some  control 

■  Must  follow  SOPs  in  place  even  if  they  don’t  make 


sense 


iPlf  Homeland 
W  Security 

Science  and  Technology 


DHS  and  DoD 

Test  &  Evaluation  Differences 


Systems  Under  Test  may  be  Tested  but  not 

Acquired  by  DHS 


•  Qualified  Product  List 

■  OT  multiple  vendors 

■  Additional  vendors  may  require  OT  later 
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iPlf  Homeland 
W  Security 

Science  and  Technology 


DHS  and  DoD 

Test  &  Evaluation  Differences 


Some  DHS  Warfighters  are  Local  First  Responders 


■  System  Assessment  or  Validation  of  Emergency 
Responder  (SAVER)  equipment 

■  Conducts  impartial,  practitioner-relevant,  operationally  oriented 

assessments  and  validations  of  commercial  off-the-shelf  equipment  that  falls 
within  the  categories  listed  in  the  DHS  Authorized  Equipment  List  (AEL). 


■  Standards/Conformity  Assessment 


■  GRaDER-  Graduated  Rad/Nuc  Detector  Evaluation  and  Reporting 
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/^Ssj |\  Homeland 
'  Security 


Conclusion 


Science  and  Technology 


Although  DHS  and  DoD  T&E  have  unique 
challenges,  both  support  the  “Warfighter’ 


Homeland 

Security 


Science  and  Technology 
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End-to-End  GPS  Multi-Platform 
Integrated  System  Testing  for 

MGUE 

Angelo  Trunzo,  Paul  Benshoof,  746th  Test  Squadron,  Holloman  AFB,  NM 
Dr.  Sultan  Mahmood,  AFMC  AAC/EB,  Eglin  AFB,  FL 
Dr.  Ray  DiEsposti,  Mitch  Markota,  Joe  Hewlett,  NAVAIR,  China  Lake,  CA 

ION  GNSS  2010 

Portland,  OR 
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Themes 


U.S.  AIR  FORCE 


•  E2E  integrated  multi-platform/multi-UE 
system  testing  for  risk  reduction 

•  Legacy  to  MGUE  transition  complexities 

•  Test  COE 

•  Test  standards 

•  Cost-effective  test  approaches,  e.g.  AWFS 

•  Joint  service  standards 
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Outline 

•  Intro  to  GPS  Modernization  and  MGUE 

•  E2E  system  testing  for  risk  reduction 

•  Test  Center  of  Expertise  (COE) 

•  Proposed  test  standards 

•  Topics  in  test  standards 

•  Testing  to  specification 

•  E2E  system  testing  for  UE  transition 

•  AJ  testing  using  AWFS  (Dr  Sultan  Mahmood) 

•  Joint  service  standards 

•  Conclusions 
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Intro  to  GPS  Modernization  and  MGUE 


•  Some  features 


-  New  signals:  LI  &  L2  M-code,  L2  C,  L5 

-  Flexible  NAV  messages 

•  Improved  ephemeris  and  clock  messages 

•  New  almanac  messages 

-  Flex  power 


New  interfaces 
>-  for  Hot  Start  of 
integrated  systems 


-  GPS  III  (LI  C,  spot  beam,  high-speed  cross  links,  integrity, 

...) 


•  MGUE 


—  YMCA  capable  Modernized  GPS  UE  which  will  eventually 
replace  legacy  and  SAASM-based  UE 


Integrated  System  Test  (1ST) 


•  SAASM  testing  emphasizes  1ST,  including  the 
SS,  CS  and  representative  sets  of  SAASM 
receivers 

-  IST2-4 

•  Similar  1ST  test  concept  is  advocated  for 
MGUE 

-  MGUE  TEMP 

•  But  —  "What  is  a  system?"  -  see  next  chart 
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E2E  System  Testing  for  Risk  Reduction 


•  End-to-End  (E2E)  system  is  defined  as  the  SS,  CS  and 
integrated  multi-platform/UE  systems 

•  Integrated  System  Test  (1ST)  should  include  testing  of 
the  functionality  of  the  interfaces  connecting 
integrated  UE  systems 

Example  of  Integrated  System 
of  Multi-Platform  and  multi-UE’s 


Host  platform  UE  supplies 
initialization  function  to 
weapon  UE 


End-to-End  Integrated  User  System 


Test  Center  of  Expertise  (COE) 


•  Led  by  the  GPSW  &  746th  TS  at  Holloman  AFB 

•  Cooperative  agreement  between  Air  Force,  Navy,  and  Army 
government  test  centers: 

-  Roles  and  responsibilities  of  test  centers 

-  Cooperation  between  the  test  centers 

—  Planning  for  efficient  use  of  limited  test  resources 

-  Identification  of  any  deficiencies  in  test  resources  and  development  of 
proposals  for  correction 

-  Development  of  test  requirements,  test  architectures,  standards, 
standardized  test  plans  and  procedures  for  cost-effective  testing 

•  The  RTO  members  of  the  COE  propose  an  E2E  testing  service 
to  the  GPSW  and  user  services 


U.S.  AIR  FORCE 


Proposed  Test  Standards 


•  Sets  of  test  documents  which  need  to  be  developed,  with  the  format  and 
content  of  each 

•  Approach  for  progressive  verification,  e.g.  developmental  and  component 
level  testing  by  UE  developers,  operational,  integrated  E2E  system  level 
testing  performed  by  government  labs 

•  Testing  approaches  for  functional,  performance  and  interface 
requirements 

•  Cost-effective  testing  approaches,  e.g.  use  of  PC  simulations,  use  of  HITL 
testing  with  GPS  simulators,  range  and  flight  testing 

•  Standardized  testing  architectures  for  different  types  of  UE 

•  How  to  test  as  an  "integrated"  system  when  various  components  are 
developed  and  available  at  different  schedules,  e.g.  making  use  of 
simulators 

•  What  performance  or  test  criteria  to  declare  a  system  as  operational 

•  Development  or  acquisition  of  test  resources 
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Topics  in  Test  Standard  Documents 


•  What  to  include  in  test  standard  documents 

•  Reuse  or  tailoring  of  existing  standards 

•  Definitions 

•  Development  of  standard  scenarios  for  testing:  how  many  scenarios,  how 
to  link  requirements  to  scenarios,  how  to  define  a  "minimal  set"  of 
scenarios  to  completely  cover  and  test  requirements  in  specification  and 
interface  documents 

•  Standardized  test  procedures 

•  Standardized  methods  to  compute  deterministic  and  statistical 
performance 

•  Design-in  of  "testability" 

•  Automated  testing  approaches 

•  Development  and  use  of  standard  test  resources,  equipment  and  facilities 


Typical  Documents 


U.S.  AIR  FORCE 


•  Test  method 

•  Diagnostic  design  specifications 

•  Manufacturing  test  requirements  design  spec 

•  Design  for  testability 

•  Test  plan 

•  Test  procedures 

•  Test  equipment 

•  Operations  and  maintenance  (O&M)  manuals 


Testing  to  Specifications 


•  Some  requirements  (e.g.  functional)  may  be  cost-effectively 
tested  with  sets  of  receivers  installed  in  racks  and  subject  to 
the  same  scenarios 

•  Some  requirements  (e.g.  performance)  are  UE  specific  so 
must  be  tested  in  real  or  simulated  operational  environment 

MGUE  Spec 


Allocation  of  spec 
requirements  for 
testing  of  many  UE 
in  parallel  in 
“equipment  racks” 
vs.  UE  specific 
test  setups 


Environmental 


V  v  —  V 

1 1 

Scenarios  &  setups  to  test  Scenarios  &  setups  to  test 

multiple  MGUE  types  in  parallel  groups  of  requirements 

f 

Test  requirements,  plans,  procedures,  etc 
(component,  lab,  E2E,  field  &  flight  tests,  etc) 


_ y _ 

Functional 


T" 

l 


_ ^ _ 

Performance 
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E2E  System  Testing  for  UE  Transition 


•  Multiple  generations  of  multi  UE  may  need  to  interface  in  an 
"integrated  system,"  including  spot  beam  capable 

•  Interfaces  for  MGUE  will  also  most  likely  change 

•  MGUE  and  all  interfaces  need  to  be  interoperable  and 
backward  compatible 


E2E  integrated  systems  need 
to  be  interoperable  and 
backward  compatible  with 
multi  generations  of  UE 


to  weapon  release 


Also  spot  beam  UE 

12 


AJ  testing  using  AWFS 


Dr.  Sultan  Mahmood 
AFMCAAC/EB,  Eglin  AFB,  FL 


Cost  Effective  Test  Approaches 


U.S.  AIR  FORCE 


*  Stand-Alone  and  Integrated  MGUE  Performance 
Testing  Under  Dynamics  and  Jamming: 

—  Lab  Testing:  Hardware-ln-The-Loop  (HITL)  Using  Antenna  Wave-Front 
Simulator  (AWFS) 

-  Van/Flight  Testing  Using  AWFS 

-  Integrated  Weapon  and  Aircraft  Testing  Using  AWFS 

•  Test  Various  Hot  Start  Data  Requirements 

•  Test  Mixed  Mode  Receiver  Operations 
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Testing  for  MGUE  Specs  and 
Inter-Operability 


•  Stand-Alone  or  Integrated  Tests 

*  Requirements  for  Standardized  Tests: 

•  Realistic  Dynamics  and  Flight  Trajectories 

•  Realistic  GPS  SV  and  Jammer  Motion,  Power  Profiles 

•  Environment  (Temp,  Vibration) 

•  EMI/EMC 

•  Realistic  Initialization  Data  for  Hot  Start,  Transfer  Alignment,  Differential 
Corrections  etc 

•  Developmental  or  Operational  Navigation/AJ  etc  Software 

•  Multiple  Host/Weapon  Receiver  Combinations 

•  Legacy 

•  SAASM 

•  M-Code  or  YMCA 

•  Multiple  Power  Levels  (Standard,  Flex,  Spot  Beam) 

•  Ability  to  conduct  Excursions,  and  What  Ifs 


Conventional  Ground  Testing  (Van) 


Frequency  Clearance,  Jammer  Scenario  Set-Up  Issues 


Navigation 
Solution 
(P,  V  etc) 


HITL- Using  AWFS 


U.S.  AIR  FORCE 


Navigation 
Solution 
(P,  V  etc) 


Measurements 


•  Simulated  GPS  and  Jammer  Signals  as  Received  by  Each  CRPA  Element 

•  CRPA  Antenna  Element  Model  Includes  Body  Masking  Effects 

•  AJGPS  System  Excited  with  RF  Signals  From  Simulated  GPS  and  Jammers 

•  Simulated  IMU  Measurements 


•  6  DOF  Generates  Actual  Weapon/Aircraft  Dynamics  and  Flight  Trajectories,  Initialization  Data 
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K*  \i 

U.S.AIR  FORCE 


Ground  and  Air  Testing  Using  AWFS 


N-Element 
CRPA 


o 


Antenna 

Wave-Front 

Simulator 

(AWFS) 


Initialization 
(or  Hot  Start)  Data 


(AJGPS) 


1 

- N 

Integrated 

GPS/INS 

- k' 

(  Nav  KF ) 

Van/Weapon/Aircraft 
Truth  Trajectory 


IMU  Measurements 


IMU 


Inputs 

Jamming  Scenario 

•  Locations 

•  Powers 

•  CRPA  Patterns 

•  Live  Satellite  Signals  Into  Actual  N-Element  CRPA 

•  IMU  subjected  to  Actual  Dynamics  vs  a  Simulated  Math  Model 

•  Jammer  Scenario  and  CRPA  Model  Used  in  AWFS 

•  AWFS  Generates  Actual  Jammer  RF  as  Received  at  Individual  Elements 

•  AJGPS  System  Excited  with  Actual  GPS  and  Jammer  Signals 


Navigation 
C>  Solution 
(P,  V  etc) 


Van/Aircraft/Lab 
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Integrated  Weapon/Aircraft  Testing 

Using  AWFS 


Inputs  for  Aircraft  Trajectory  Gps  +  Jamming  signals 


•  Powers 

•  Aircraft  CRPA  G/P 


Navigation 
Solution 
(P,  V  etc) 


Initialization 
(or  Hot  Start) 
Data 


Aircraft 
to 

Weapon 
Interface 

GPS  & 

Jamming  Signals 


•  Live  (Captive)  Tests  Require  Two  AWFSs 

•  Lab/HITL  Can  Be  Conducted  Using  a 
Single  AWFS 

•  Weapon/Aircraft  Interfaces  Tested  in 
Operational  Environments 


Inputs  for  Weapon  Trajectory 


GPS  Constellation 

•  Almanac 

•  Power 

•  CRPA  Patterns 

•  Error  Sources 


Jamming  Scenario 

Locations 

Powers 

Weapon  CRPA  G/P 


Antenna 

Wave-Front 

Simulator 

(AWFS) 
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Conclusions 


U.S.  AIR  FORCE 


•  Integrated  System  Test  (1ST)  should  include  testing  of 
interfaces  and  generations  of  UE  multi- 
platform/multi-UE  integrated  systems 

•  GPS  Test  Center  of  Expertise  (COE)  offers  a  means  to 
coordinate  and  manage  the  large  test  effort  needed 

•  Need  test  standards! 

•  Need  cost  effective  test  approaches,  e.g.  AWFS 

•  Recommend  Joint  Service  Standards 
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Integrated  Test: 
Challenges  &  Solutions 


Honorable  Dr.  Michael  Gilmore 
Director,  Operational  Test  &  Evaluation 
Presentation  to  NDIA 
March  15,  2011 


Outline 


•  DOT&E  Initiatives 

•  Integrated  Test 

•  Challenges  to  Integrated  Test 

•  Integrated  Test  Solutions 

•  Design  of  Experiments  and  Integrated  Testing 

•  Conclusions 


DOT&E  Initiatives 


Integrated  Testing 

Developmental 
Operational 
Live  Fire 

Use  Scientific  Test  Design 
(e.g.  DOE  ) 


Testers  Engage 
Early 

Testable,  Mission- 
oriented 
Requirements 


Field  New 
Capabilities 
Rapidly 

Accelerated  Testing 


Improve 

Suitability 

Reliability  Growth 
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Integrated  Test 


•  What  is  Integrated  Testing? 

-  A  cohesive  test  and  evaluation  plan  that  spans  all  stages  of 
testing. 

-  Integrated  test  is  NOT  simply  combining  data  from  different  test 
events. 

-  Integrated  test  is  NOT  a  replacement  for  dedicated  OT. 

•  Integrated  Test  methods: 

-  Using  data  from  CT,  DT,  and  OT  to  inform  the  next  stage  of  testing 

-  When  appropriate,  combine  CT,  DT,  and  OT  data 

•  Reduce  test  time,  increase  statistical  confidence  and  power 

-  Integrate  DT  and  OT  test  objectives 

•  Enhance  operational  realism  in  DT  to  reduce  OT  requirements 

-  Design  of  Experiments  helps  plan  efficient,  integrated  testing 

•  Plan  testing  as  a  sequence  of  tests 


Integrated  Test  Can  Be  A  Challenge 


•  Not  business  as  usual 

-  Unclear  responsibilities.  Who  is  in  charge  of  the  test? 

•  Contractual  issues 

-  Limited  access  to  contractor  test  data  and  test  procedures 

•  DT  and  OT  test  objectives  conflict 

-  Combining  tests  maybe  impossible 

•  Combining  data  maybe  irresponsible 

-  How  the  test  is  executed  affects  results 

—  How  the  system  design  evolves  affects  results 

•  Late  involvement  of  OT  testers 


Affects  all  of  the  above 


Integrated  Testing  Makes  Sense! 


•  Enables  efficient  testing 

—  OT  assessments  can  take  advantage  of  CT  and  DT  data 

•  Assessing  system  performance  as  the  design  matures 
requires  consolidation  of  data 

—  e.g.,  reliability  growth 

•  System-of-systems  requiring  coordination  of  multiple 
test  programs  are  increasingly  common 

•  Discovery  in  OT  is  expensive 

-  We  need  to  find  problems  early  in  DT 

•  Design  of  Experiments  facilitates  efficient,  integrated 
testing. 


Integration  of  Available  Data 


Ballistic  Missile  Defense 
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Ballistic  Missile  Defense 


•  Motivation:  Estimate  system  effectiveness  with  small  sample  sizes 

•  Probability  of  Success  (PES)  is  the  probability  of  successfully  negating  a 
ballistic  missile  threat  using  the  Ballistic  Missile  Defense  System  (BMDS) 

•  Traditional  probability  based  approaches  are  data  intensive 

-  Conditional  probability  model  requires  lots  of  data  in  each  stage 


Track 


PES 


Launch 


Detect 


Engage/Kill 


PES  for  Ballistic  Missile  Defense 


DOT&E  turned  probability  problem  into  sampling  problem 

-  PES  =  (#  Kills)/(#  Launches) 

-  PES  =  (#  Kills)/(#  Detections)  •  (#  Detections)/(#  Launches) 

-  PES  =  (#  Kills)/(#  Tracked)  •  (#  Tracked)/(#  Detections)  •  (#  Detections)/(#  Launches) 

-  ...  repeat ... 


Launch 

Detect 

Track 

... 

Intercept 

Kill 

Test  1  xx 

XX 

Test  2 

XX 

XX 

XX 

Test  3 

XX 

XX 

XX 

XX 

XX 

i - 

XX 

Test  4 

XX 

XX 

XX 

XX 

o  < - 

Test  5 

XX 

XX 

XX 

XX 

XX 

XX 

Partial 

Tests 

Failure  at 
Intercept  Stage 


•  DOT&E  PES  methodology  applied  to  Patriot  data 

-  Produces  similar  results  to  traditional  analysis  for  large  datasets  (validates  method) 

-  Validation  indicates  that  the  similar  results  were  achieved  with  less  data 

•  DOT&E  PES  methodology  applied  to  Aegis  BMD  (smaller  dataset) 

-  Refines  the  results  from  simple  success/failure  analysis  to  account  for  partial  tests 

-  Results  included  in  DOT&E  Report  to  Congress 


Maximize  use  of  data  from  relevant  test  events 


Integrated  Testing  for  Reliability 


ANSI/GEIA-STD-0009 


1.  Understand  user  requirements  and  constraints 

-  Reliability  requirements  include  the  anticipated  use  environment 

2.  Design  for  Reliability  (DFR)  and  Re-design  for  Reliability 

-  This  means  that  user  needs  will  be  allocated  through  system  model  to  reliability 
specifications  at  lowest  component  levels. 

-  Lowest  level  reliability  specifications  include  internal  stresses  and  impacts  of  use 
environment 

-  Redesign  as  needed  to  meet  allocated  reliability  requirements 

3.  Produce  reliable  systems 


During  DT,  all  sub-assemblies,  components,  etc  should  demonstrate  required 
reliability  in  anticipated  use  environments 

Meeting  reliability  requirements  will  often  require  reliability  growth  programs  for 
components  utilizing  repeated  DT  experiments 


4.  Monitor  and  assess  user's  experienced  reliability 
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Stryker  NBCRV  Design  For  Reliability 


1.  Production  Verification  Testing  (PVT)  was  halted  prematurely  due  a  large 
number  of  System  Aborts 

•  Did  not  meet  the  user  requirement  of  1000  Mean  Miles  Between  System  Aborts 
(MMBSA)  for  the  base  vehicle 

•  No  reliability  requirement  for  NBC  sensors 

2.  System  contractor  implemented  Design  For  Reliability  to  improve  base 
vehicle  reliability  (2007-2008) 

3.  NBCRV  underwent  8000  mile  Reliability  Growth  Test  (RGT)  in  2009  to 
determine  whether  reliability  had  improved. 

•  Base  vehicle  reliability  dramatically  improved  over  PVT  (2000  MMBSA). 

•  Little  change  in  NBC  sensor  reliability. 

4.  Dramatic  improvement  in  reliability  between  PVT  and  RGT  but  no  reliability 
growth  seen  during  RGT  itself. 

5.  Requirements  drove  the  focus  of  DFR,  but  requirements  addressed  only  the 
base  vehicle  and  not  the  NBC  sensors 

6.  DFR  is  a  powerful  tool  to  improve  reliability,  but  must  address  entire  system 
to  be  effective 


12 


Integrated  Testing  for  System  of  Systems 

Air  Warfare  Ship  Self-Defense  Enterprise 


Radars:  SPS-49,  SPS-48, 
SPQ-9B,  MFR... 


CIWS  search 

fS,  IRST 

SLQ-32 
DEW 


Ship  Defense  MOE 

Probability  of  Raid  Annihilation  (PRA) 

is  the  probability  a  particular  stand-alone  ship,  as  a  system  of  systems, 
will  defeat  a  raid  of  X  cruise  missiles  arriving  within  Y  seconds 


—  NATO  Seasparrow, 

ESSM 


Onboard  EA 


CEC 


Passive  countermeasures 


MK  214 
Chaff 


c  g 


Multi-threat  raid 


MK216 

Chaff 


Battle 


Timeline 


&  30  seconds 


I 


Battle 

Space 


0-12  nmi 


Air  Warfare  Ship  Self-Defense  Enterprise 


•  Combat  systems  for  aircraft  carriers  and  amphibious  ships 
composed  of  systems  from  various  program  offices 

-  Previously,  each  program  office  developed  its  own  test  program 

-  Each  test  program  focused  on  an  individual  system,  not  on  the  integrated 
combat  system  or  the  overall  air  defense  mission 

•  Ship  Self-Defense  Enterprise  coordinated  these  various  test 
programs 

-  Provides  significantly  better  end-to-end  testing  of  the  integrate d  combat 
system,  focusing  on  the  air  self-defense  mission 

-  Used  principles  of  Design  of  Experiments  to  develop  test  plan 

•  For  air  self-defense,  the  Navy  estimates: 

-  Before  Enterprise,  testing  cost  about  $1.1  Billion  FY05  through  FY15 

-  Enterprise  saved  $240  Million  out  of  $1.1  Billion 


Better  testing  for  less  money 


Integrated  Testing  to  Avoid  Late  Problem 

Discovery 


Integrated  Defensive  Electronic  Countermeasures  (IDECM) 

& 

Miniature  Air  Launched  Decoy  (MALD) 


Late  Discovery  of  Problems 
IDECM  and  MALD 


•  Limited  operational  realism  in  early  testing 

-  IDECM  -  use  of  special  DT  equipment  to  reduce  test  costs 

-  MALD  -  no  long-duration  carriage  of  decoys 

•  Significant  problems  discovered  in  IOT&E 

-  IDECM 

•  Uncommanded  deployments  and  problems  severing  decoys 
created  safety  problem  for  ground  crew 

•  Intermittent  failures  resulted  in  decoys  being  prematurely 
discarded  and  in  poor  reliability 

-  MALD 

•  Long-duration  flight  caused  premature  failures  when  decoys 
were  launched. 


Design  of  Experiments  (DOE) 


•  A  method  for  planning  efficient  integrated  testing. 

•  For  integrated  testing,  DOE  can  inform: 

-  Plan  testing  as  a  sequence  of  tests 

-  Screen  out  insignificant  factors  in  DT  to  focus  OT 

-  Control  factors  in  DT  that  are  difficult  to  control  in  OT 

-  Split  factors  across  test  periods 

-  Ensure  that  operational  envelope  is  covered 

•  DOE  is  an  Industry  Best  Practice 

-  DOE  traditionally  applied  in  DT  context,  but  we  are  seeing  great  gains 
using  the  methodology  in  integrated  testing  and  operational  testing 

•  Example  of  DOE  in  DT:  wind  tunnel  testing 

-  Characterize  the  aerodynamic  behavior  of  the  X-31  Enhanced  Fighter 

-  Traditional  techniques  would  require  1000  +  test  points 

-  DOE  applied  &  testers  were  able  to  characterize  aerodynamic 
performance  in  104  test  points. 


Example  of  Integrated  Testing  Employing  DOE 
Joint  Chemical  Agent  Detector 


•  Problem:  Agents  are  unable  to  be  tested  in  an  OT. 

-  Agent,  temperature,  water  vapor  content,  operating  mode  and  agent 
concentration  were  systematically  varied  in  DT  using  a  Response  Surface 
Design. 

-  Allowing  for  operational  factors  affecting  performance  to  be  assessed  in 
OT  (Service,  environment,  and  mission  tactics) 

JCAD 


Conclusions 


•  Efficient  integrated  testing  is  a  must. 

•  Integrate  Test  solutions  are  as  unique  as  the  challenges 

-  Plan  CT  and  DT  tests  to  enable  OT  use  of  the  data. 

-  Assessing  system  reliability  requires  integrated  test. 

-  System-of-systems  requires  integration  of  multiple  test  programs. 

-  Operational  realism  in  DT  allows  problems  to  be  discovered  early 

•  Key  Ingredients  for  Integrated  Testing 

-  Early  engagement  of  Operational  Testers 

-  Robust  data  collection  and  documentation 
—  Experimental  Design 

•  Can  help  ensure  integrated  testing  is  comprehensive 

•  Provide  confidence  and  power  across  the  operational  envelope 


Every  Program  and  every  challenge  has  a  unique  solution  to  Integrated  Testing 


Backups/Extras 


Integrated  Testing  for  Reliability 


Design  for  Reliability 
& 


Reliability  Growth 
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Stryker  NBCRV 

Design  For  Reliability  Case  Study 


•  Reduced  Risk:  "...  program  has  recently 


GENERAL.  DYNAMICS 

LdKl  Syren'S 


Engineering  Design  &  Development 

Compliance  Execution  Conmnuous  Improvemeni 


undergone  its  DFR  phase,  after  which  it 
demonstrated  a  four  times  improvement  in 
reliability." 

•  Reduced  Acquisition  Time: "... 
subsequent  reliability  testing  was  cut  almost 
in  half  since  the  vehicles  demonstrated  the 
required  level  of  reliability." 

•  Reduced  Cost:  "...the  amount  saved  from 
early  discontinuation  of  the  test  was  greater 
than  the  spending  on  the  DFR  phase  by 
almost  3  times." 

The  Cost  AND  Schedule 
Optimal  Solution  is  to 
Design  for  Reliability 


Design  For  Reliability  Implementation  and  Verification  at  GDLS  -  Continued 

Based  on  a  recent  Stryker  program  development  we  nave  learned  that  DFR  not  oriy  drastically  increased  demonstrated 
reliability  but  also  dramatically  saved  developmental  costs,  especially  during  Developmental  Test  The  Stryker  NBCRV 
program  has  recently  undergone  its  DFR  phase,  after  which  i;  demonstrated  a  four  times  improvement  in  reliability  The 
subsequent  reliability  testing  was  cut  almost  w  half  since  the  vehicles  demonstrated  the  required  level  of  reliability  with 
confidence  It  is  imoortant  to  note  that  the  amount  saved  from  early  discontinuation  of  the  test  was  greater  than  the 
spending  on  the  DFR  phase  by  almost  3  times  This  experience  debunks  the  notion  that  there  must  be  a  tradeoff  between 
reliability  and  faster  fielding  Our  external  customers,  at  times  must  be  convinced  of  the  importance  of  the  proper  Reliability 
Program  using  the  DFR  toolbox  for  the  bottom-line  cost  saving  Any  lack  of  attention  to  reliability,  dunrg  design  stage,  and 
absence  of  the  Design  For  Reliability  methodology  wHI  lead  to  inadequate  performance  of  the  product  during  system  testing 
and  exceedingly  high  expenditures  before  fielding  the  system.  All  of  that  can  be  avoided  if  a  proper  amount  of  attention  and 
investment  is  allocated  to  reliability  as  early  as  possible.  The  Systems  Engineering  Master  Plan  (SEMP)  explicitly  addresses 
employment  of  the  DFR  tools  and  this  is  the  plan  to  follow  Proper  language  in  a  Request  for  Proposal  (RFP)  and  contracts 
that  utilizes  recent  ANSI/GEIA-STD-0009  car  help  to  shift  the  focus  of  the  development  towards  designing  and  then 
producing  reliable  and  robust  defense  systems  it  was  not  unnoticed  that  the  Ground  Combat  Vehicle  (GCV)  RFP  -used  0009 
language  and  required  a  viable  reliability  program  up-front  m  the  design  process 


GDLS  R-Plan  &  Process 


POD  -  GEtA-STD-0009 


3  Reliability 
Analysis 

4  Reliability 
Tesbng 

5  Supply  Chain 
Management 

6.Failure 
Tracking  and 
Reporting 

/.Verification 
and  Validation 

8. Reliability 
Improvements 


Source:  Ruma,  J  and  Tananko,  D,  Design  For  Reliability  Implementation 


and  Verification  at  GDLS,  Around  Edge  -SE,  10/11/01 


By:  Jim  Ruma 
Dmitry  Tananko 


Vxr«  ECAO  •  S8  •  •a-i-31 
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Reliability  growth  depends  on 
two  distinct  reliability  models 


•Full  system  model  guides  integrated  testing. 

•Provides  an  initial  guess  at  system  reliability 

•The  goal  is  NOT  to  create  a  complete  model  of  what  will  fail  when  and  why 


Full  system  model  used  to  allocate  system 
reliability  down  to  required  reliability  at 
lowest  levels. 
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PM 2  RELIABILITY  GROWTH  PLANNING  CURVE 
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Growth  model  used  to  track  and  predict 
reliability  of  individual  pieces(sub- 
systems,  etc)  in  DT/IT  and  of  full  system 
in  IT/OT 
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As  design  matures... Reliability  Growth 
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•Senior  Leadership  Participation 


-  Government  &  Industry  Support 
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Participants 


STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


ATK 


GENERAL  DYNAMICS 

Land  Systems  _ 

cubic  wyle 


DEFENSE  APPliCATXOfJS 


Raytheon 

THE  O’BRYON  GROUP 

James  F.  O’Bryon  *  National  Defense  and  Security  Consultant 


© 

A  L  I  O  N 

SCIENCE  AND  TECHNOLOGY 


BAE  SYSTEMS 


Drexcl 

consulting  group 


GRUMMAN 


GENERAL  DYNAMIC! 


NAVISTAR' 

DEFENSE 


An  employee -Owned  Company 


ICOTE  Mission 


Provide  forum  for  senior  Test  and  Evaluation  representatives 
from  Defense  Department  &  U.  S.  defense  system  manufacturers 
to  meet  and  review  issues  of  common  interest  and  concerns. 
Discuss  T  &  E  policies  and  procedures  impacting 
weapons  systems  development,  test,  procurement  and  use. 

Takeaways 

•  Discuss  &  Gain  Feedback 
•  OSD  Policies  and  Emerging  Issues 
•  ICOTE  Cooperation  to  Benefit  Warfighters 

•  Topics  of  Interest 


Major  T&E  Web  Sites 

OSD  http://www.dote.osd.mil/pub/otherrep.html 

OSD  http://www.acq.osd.mil/dte/pg/index.html 

Army  http  ://www.  atec.  army,  m  i  l/i  mages/ATEC_F  I N  AL_20 1 00929_low.  pdf 

Air  Force  http://www.afotec.af.mil/index.asp 

Marines  http://www.marines.mil/unit/hqmc/mcotea/Pages/index.aspx 

Navy  http  ://www.cotf .  navy,  m  i  l/i  ndex.  htm 

NDIA  http://www.ndia.org/DIVISIONS/INDUSTRIALWORKINGGROUPS/ 

INDUSTRIALCOMMITTEEONTESTANDEVALUATION/Pages/default.aspx 


ICOTE  Initiatives 

Include  Systems  Engineering  and 
Developmental  Test  Disciplines 

Emphasis  Continues  on  Reliability 

Systems  Focus  on  Survivability, 
Effectiveness  and  Suitability 

Ship  Sub  Committee 

Testing  of  Cyber 

Field  New  Capability  Rapidly 

Engage  Early  to  Improve  Requirements 


Test  and  Evaluation 


“Serving  the  Warfighter  in  a  Cost 
Constrained  Environment” 


16  March  2011 


Mr.  Chris  DiPetto 

Office  of  the  Deputy  Assistant  Secretary  of  Defense 
for  Developmental  Test  and  Evaluation 


DoD  Budget  Realities 


•  Defense  base  funding  must  have  real  growth  to  sustain  force  structure  and 
modernization 

•  Fighting  Two  Wars 

•  Confronting  ongoing  terrorist  threats  around  the  globe 

•  Facing  major  powers  investing  heavily  in  their  military 

•  Requires  2-3%  real  growth 

•  The  current  and  planned  base  defense  budget  has  steady,  but  modest  growth  of  1%  per  year 

•  To  preclude  reductions  in  needed  military  capability,  the  difference  of  1-2%  a  year  will  be  made 
up  elsewhere  in  DoD 

•  In  May,  SecDef  began  a  hard,  unsparing  look  at  how  DoD  is  staffed, 
organized,  and  operated 


“...in  May  I  called  on  the  Pentagon  to  take  a  hard  and  unsparing  look  at  how  the 
department  is  staffed,  organized  and  operated.  I  concluded  that  our  headquarters  and 
support  bureaucracies,  military  and  civilian  alike,  have  swelled  to  cumbersome  and 
top-heavy  proportions,  grown  over-reliant  on  contractors  and  grown  accustomed  to 
operating  with  little  consideration  to  cost.”  ....Secretary  of  Defense  Robert  M.  Gates 
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enter  the  SecDef  Plan  for 
Efficiency 


•  Target  Affordability  and  Cost 
Growth 

•  Incentivize  Productivity  & 
Innovation  in  Industry 

•  Promote  Real  Competition 

•  Improve  Tradecraft  in  Services 
Acquisition 

•  Reduce  Non-Productive 
Processes  and  Bureaucracy 


(L)  Secretary  of  Defense  Robert  M.  Gates 
(R)  USD  AT&L  Dr  Ashton  B.  Carter 


“Consumers  are  accustomed  to  getting  more  for  their  money  -  a  more  powerful 
computer,  wider  functionality  in  mobile  phones  -  every  year.  When  it  comes  to  the 
defense  sector,  however,  the  taxpayers  had  to  spend  significantly  more  in  order  to  get 
more.  We  need  to  reverse  this  trend.”  ....Secretary  of  Defense  Robert  M.  Gates 


Look  at  Acquisition? 


Cost 


THE  USUAL  (?)  SUSPECTS 


Over  Budget 

-  GAO:  96  MDAPs,  $300B  over  initial  estimates 

Schedule 

Late  to  Need 

-  Getting  capability  to  the  user  to  meet  urgent  needs 

Performance 

Programs  failing  Operational  Test 

-  Suitability  issues 

-  Late  discovery  of  failure  modes 

-  Performance  shortfalls 

-  Interoperability 
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Where  Can  the  T&E  Community  Find 


•  Reduce  the  amount  of  testing 

(manage  our  appetite,  don't  “test  for  testing’s  sake”) 

-  What  is  an  “adequate”  amount  of  testing  to  support  the  required  evaluation? 

•  Increase  the  efficiency  of  testing 

(e.g.,  lower  cost  per  test  point  or  knowledge  gained) 

-  Tools  Design  of  Experiments  (DOE) 

-  Capital  Utilization 

-  M&S 

•  Use  T&E  as  a  means  to  reduce  the  cost  of  acquisition 

(the  “test-earlier”  part) 

-  Discovering  failure  modes  early 

-  Fully  inform  decision  makers 


Panel 


Bios 


Biographies 


Mr.  Jack  Manclark,  Air  Force  T&E  Executive 

Mr.  Manclark  (Jack)  is  the  Director  of  Test  and  Evaluation, 
Headquarters  U.  S.  Air  Force  in  Washington  D.C.  He  is  responsible  for  all 
Air  Force  policy,  resources  and  oversight  of  developmental  and 
operational  testing,  and  is  the  focal  point  for  foreign  material  acquisition 
and  exploitation. 


Mr.  David  AC.  Grimm,  Acting  Director  Deputy  Under  Secretary  of  the 

Arm y,  T&E  Office 

Mr.  Grimm  (Dave)  is  the  acting  Assistant  Deputy  Under  Secretary  of 
the  Army  for  Test  and  Evaluation.  He  is  responsible  for  all  Army 
Acquisition  Category  (AC AT)  I  and  II  programs  and  the  Chemical 
Biological  Defense  Program.  He  serves  as  the  integrator  and  primary 
agent  for  the  Secretary  of  the  army  in  coordinating  T&E  issues,  positions 
and  reports  with  the  other  Military  Departments,  the  Office  of  the 
Secretary  of  Defense,  the  Joint  Staff  and  Congress. 


Biographies 


Dr.  Steve  Hutchison ,  DISA  T&E  Executive 

Dr.  Hutchison  (Steve)  assumed  the  duties  as  the  Test  and  Evaluation 
(T&E)  Executive  in  August  2005,  to  oversee  strategic  planning,  resourcing, 
and  execution  of  the  T&E  mission  for  the  Agency,  and  represent  DISA  to 
the  DoD  T&E  community.  Dr.  Hutchison  supervises  the  activities  of  the 
Joint  Interoperability  Test  Command  and  the  Office  of  the  T&E  Executive. 
Prior  to  his  arrival  in  DISA,  Dr.  Hutchison  served  in  the  office  of  the 
Director,  Operational  Test  and  Evaluation  (DOT&E)  and  the  Army  Test  and 
Evaluation  Command  (ATEC). 

Ms.  Amy  Markowich,  Navy  T&E  Executive 

Ms.  Markowich  (Amy)  currently  serves  as  the  Deputy,  Department  of 
the  Navy  Test  and  Evaluation,  Executive  Office  of  the  Assistant  Secretary 
of  the  Navy  (Research,  Development  and  Acquisition  (RD&A).  In  this  role 
she  is  responsible  for  the  integration  of  Test  and  Evaluation  (T&E)  across 
the  Navy  and  Marine  Corps,  enhancing  the  T&E  workforce  and 
infrastructure,  and  ensuring  complete  adequate  testing  to  demonstrate 
suitable  and  effective  operations  in  the  joint  battle  space. 


Biographies 


Mr.  Tom  Wissink,  Director  of  Integration,  T&E,  Lockheed  Martin 

Mr.  Wissink  (Tom)  is  the  Lockheed  Martin  Corporate  Engineering  & 
Technology  Director  of  Integration,  Test  &  Evaluation  and  a  Corporate 
Senior  Fellow.  He  has  worked  in  system/software  integration  and  test, 
software  development,  and  configuration  management  for  more  than  35 
years. 


“s/v 

A  Combat  Support  Agency 


Efficiency  Initiatives 


•  IT  Acquisition  Reform 

-  fully  integrated  test,  evaluation,  and  certification 

•  Federate  DoD  Capabilities  -  "Joint  IT  Range" 

-  find,  connect,  collect,  release 

•  T&E  network  convergence 

-  common  DoD  T&E  network 

•  Test  as  a  Service 

-  enterprise  Test  Tools 

•  Virtualization 

-  move  to  cloud  and  virtual  testing  concepts 

-idsi 


Continuous  Integration  and  Test 

1  Developmental 
’  Operational 


Voice  o<  the  User 


•  Operational 

•  Interoperability 

•  Security  V 

**'r**\ 

14-8  Weeks  | 
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(Capability 
Oevelo 


User  Requirements 


Working  Product 

(Militarily  lhahil  Capability  | 


Making  Acquisition  Processes  and  Infrastructure 
Responsive  to  the  Warfighter 


NDIA  T&E  Conference  Plenary  Panel 

T&E:  Serving  the  Warfighter  in  a  Cost  Constrained 
Environment  --  Army  Perspective 


Acting  Assistant  Deputy  Under  Secretary  of  the  Army 

(Test  and  Evaluation) 

Mr.  David  K.  Grimm 
16  March  2011 


isition? 


Some  Risk  Factors... 

•  Technology 

•  Integration 

•  Program 
(cost,  schedule, 
performance) 


Army  T&E:  Serving  the  Warfighter 
in  a  Cost  Constrained  Environment 


Army  is: 

*  Synchronizii 
Integrated  N< 
Effort 


Driving  TSEjnvestment 
Collaboration 

Leveraging  BRAC  Relocations, 
Organizational  Realignments, 
Process  /  Cultural  Shifts 

Integrating  Tests  w  /  Training  & 
Readiness  Exercises 
Supporting  Competitive 
Prototyping  and  All  Available 
Data  Sources 


Context:  Army  Testing  in  a  Time  of  War 
“The  Warfighter  Needs  to  Know” 


Developmental  Test  Reports 
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Operational  Test  Reports 


OT 


Evaluations  &  CLRs 
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Sample  of  Army  Test  „Bat1e  Rhythm” 
Week  of  24  APR  -  01  MAY  2009 

1 ,299  Active  Test  Projects 

•  129  Rapid  Fielding  Events 

•  237  Joint  Tests  /  Projects 

•  663  Non-Eval  Program  Tests 

•  270  MDAP  “PORs” 


Mandated  by 
DoDI  5000.02 


Responsive  and  Agile  IT  T&E 


Tampa,  FI 

March  16,2011 
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National  Defense  Industrial  Association 


Sources  of  Stakeholder  Recommendations 


•  Defense  Science  Board 

-  Developmental  Test  and  Evaluation,  May  2008 

-  DOD  Policies  and  Procedures  for  the  Acquisition  of  Information 
Technology,  March  2009 


•  2010  National  Defense  Authorization  Act  (Sec  804) 

-  Implementation  of  a  New  Acquisition  Process  for  Information 
Technology  Systems 

•  NDIA-OUSD  (AT&L)  System  Engineering  Division/Development  T&E 
Committee  and  Software  Industry  Experts 

-  Software  T&E  Summit/Workshop  September  2009 

-  Joint  Authored  White  Paper,  Dec  2009 


National  Academies  of  Science  Study 

-  Achieving  Effective  Acquisition  of  Information  Technology  in  the 
Department  of  Defense,  December  2009 


National  Defense  Industrial  Association 


DOD  Agile  IT  Precepts 


1)  achieve  significant  time  and  cost  resource  efficiencies 

2)  support  software  application  “sprints” 

3)  provide  tailored  test  environments  established  on 
demand 

4)  create  a  virtual  library  of  systems  and  services  to  avoid 
having  to  stand  up  physical  systems  for  every  test 

5)  establish  a  DOD  wide  accepted  restructured  IT  T&E 
process 
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Panel  Members 


Panel  Chair  Dr  Steven  Kimmel 

Senior  VP  Alion  Science  and  Technology 

Interop/Network  Cert/Assurance  Dr  Steven  Hutchison 

Test  and  Evaluation  Executive,  DISA 

Dr  James  Streilein 

Dep  Dir  DOT&E  (Comm  &  Space  Systems) 

Ms  Darlene  Mosser-Kerner 
Dep  Dir,  DT&E  (AT&L/DDRE) 

Mr  Eustace  King 
DIACAP  Tiger  Team 


OT&E 

DT&E 

OSD/NII 
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National  Defense  Industrial  Association 


Panel  Introduction 


Approach 

Engage  with  the  AUDIENCE  to  understand  and  explore  the 
community  of  challenges,  issues  and  solutions  via  Q  &  A’s  vice 

death  by  ppt. 


GOAL 


Achieve  an  integrated  T&E  responsive, 
effective  and  efficient 
(mission-focused  on-demand) 
agile  capability 


y  -am  Usa  PS^t  % 


Panel  Focus 


Customer  complaint:  IT  testing  is  a  serial  process  that  costs  too 
much  and  takes  too  long  to  complete. 


The  Challenge: 

1)  How  to  integrate  users  and  testers  such  that  a 
common  set  of  standards  can  address  joint 
interoperability  and  information  assurance  testing 
into  an  agile,  mission  focused  team. 
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Panel  Focus 

Customer  complaint:  IT  testing  is  a  serial  process  that  costs  too  much 
and  takes  too  long  to  complete. 

The  Challenge: 

1 )  How  to  integrate  users  and  testers  such  that  a  common  set  of  standards 
can  address  joint  interoperability  and  information  assurance  testing  into 
an  agile,  mission  focused  team. 


How  to  network  DOD  testing  capabilities  so  that  we 
are  able  to  test  things  in  parallel,  i.e.,  establish  a 
realistic  test  environment  whereby  one  test  affords 
sufficient  &  thorough  “stakeholder”  data 
collection 
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Panel  Focus 

Customer  complaint:  IT  testing  is  a  serial  process  that  costs  too  much 

and  takes  too  long  to  complete. 

The  Challenge: 

1 )  How  to  integrate  users  and  testers  such  that  a  common  set  of  standards 
can  address  joint  interoperability  and  information  assurance  testing  into 
an  agile,  mission  focused  team. 

2)  How  to  network  DOD  testing  capabilities  so  that  we  are  able  to  test  things 

in  parallel,  i.e.,  establish  a  realistic  test  environment  whereby  one  test 
affords  sufficient  &  thorough  “stakeholder”  data  collection 

How  stovepipe  test  beds  be  melded  together  to 
enable  testers  to  locate  needed  assets,  connect 
them  into  the  test,  collect  the  data  needed,  then 
release  the  asset  when  the  test  has  been 
completed. 
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Opening  Panel  Question 


What  are  the  impediments  to  achieve  a  “virtual” 

network  that  can  satisfy  a  “parallel”  efficient  IT  T&E 
process? 


9 


Panel  Question  #  2 


National  Defense  Industrial  Association 


What  are  the  technical,  policy  or  procedural  obstacles 
that  need  to  be  overcome  to  - 

1)  achieve  an  operationally  realistic  environment  whereby  IT 
test  data  can  be  shared  across  DT-OT-IA-certifications  and 
accreditation? 

2)  “test  by  one,  accept  by  all”,  e.g.,  IT  T&E  reciprocity 
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Closing  Comments 


*  IT  systems  are  different  than  weapons 
systems... current  DOD  5000  inappropriate  for  both 

•  DOD  agile  developed  IT  systems/applications  are  on 

the  horizon . IT  “sprints”  projects  will  cause  closer 

collaboration  between  users,  developers  and  testers 


“Soda  straw”  or  serial  T&E  to  be  replaced  by  parallel 
acceptance,  certification,  accreditation, 
interoperability  and  integration-  test  once,  accept  by 
all. 
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X 


PS*£.  Q» 


Information  Systems  Summit  II 

“What’s  All  This  Agile  Stuff  About, 
Anyway?” 


April  4-6,  201 1 
Event  #1750 
Hyatt  Regency  Baltimore 
Baltimore,  MD 
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UNCLASSIFIED 


Operational  Test  & 
Evaluation  Overview 


HQ  USSOCOM  J8-0 
LTC  Kevin  Vanyo 
16  March  2011 


The  overall  classification  of  this  briefing  is: 
UNCLASSIFIED 


UNCLASSIFIED 


Agenda 


UNCLASSIFIED 


OT&E  Authority 

Mission  and  Tenants 

Responsibilities 

Operationally  Effective, 
Suitable,  &  Safe 

Documentation 

Environment 

T&E  Implementation 

Examples 

Conclusion 


UNCLASSIFIED 


J8-0 
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UNCLASSIFIED 


Authority 


Concurrent  with  Title  10  USC  Authority  and  Head 
of  Agency  responsibilities,  USSOCOM  ensures 
that  the  systems,  products,  and  equipment  fielded 
to  Special  Operations  Forces  (SOF)  are 
operationally  effective,  suitable,  and  safe.  These 
assurances  are  gained  through  the  test  and 
evaluation  process. 


USSOCOM  Directive  71-5,  Operational  Test  and  Evaluation 
10  USC  139 


UNCLASSIFIED 


J8-0 
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UNCLASSIFIED 


Mission 


WHAT  WE  DO 


Ensure  that  USSOCOM  Acquisition  Program  System 
Capabilities,  Independently  Evaluated,  are  Measured 
Against  Validated  Requirements  Under  Realistic 
Operational  Conditions  to  Determine  a  System’s 
Operational  Effectiveness,  Suitability  and  Survivability 
Prior  to  Fielding  to  Special  Operations  Forces 


UNCLASSIFIED 


J8-0 
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T&E  Tenants 


UNCLASSIFIED 


UNCLASSIFIED 


J8-0 
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UNCLASSIFIED 


T&E  Tenants  -  “Flexibility” 
Operational  Test  Agencies 


SOCOM 

ARMY 

NAVY 

AIR  FORCE 

MARINE  CORPS 

All  Services: 

AFOTEC 

ATEC 

COMOPTEVFOR 

MCOTEA 

Army  Test  & 
Evaluation 
Command 
(ATEC) 

Commander, 
Operational  Test  & 
Evaluation  Force 
(COMOPTEVFOR) 

Air  Force 

Operational  Test  & 
Evaluation  Center 
(AFOTEC) 

Marine  Corps  Test  & 
Evaluation  Activity 
(MCOTEA) 

JITC 

National 

Assessment  Group 
(NAG) 

JITC 

JITC 

JITC 

COMOPTEFOR 

JITC 

USASOC  Test  & 
Evaluation  Division 


(TED) 


18th  Flight  Test 
Squadron  (AFSOC) 

MCPD,  NAVSEA 
NAVAIR,  SPAWAR, 
WARCOM 


UNLIKE  MOST  OF  THE  SERVICES, 


USSOCOM  HAS  MULTIPLE  SOU 

FOR  OT&E 


UNCLASSIFIED 


J8-0 
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UNCLASSIFIED 

J8-0  Responsibilities 


■  Oversee  OT&E  for  SOF  Acquisition  Programs 

■  Develop  and  Implement  OT&E  Policy  via  USSOCOM  Directive  71-5 

■  Assess  and  Determine  system: 

>  Effectiveness 

>  Suitability 

>  Interoperability 

>  Safety 

■  With  J4  and  PEO,  issue  a  System  Production  Certification  (SPC)  and/or 
Fielding  and  Deployment  Release  (F&DR) 

■  Review  Requirement  Documents  for  Relevancy  and  Testability 

■  Assist  /  Approve  the  Test  Strategy  in  the  Single  Acquisition  Management 
Plan  or  the  Test  and  Evaluation  Master  Plan 

■  Coordinate  w /  Program  Manager  in  Selection  of  the  Operational  Test  Agency 

■  Observe  Critical  Operational  Test  Activities 

■  Validate  New  Equipment  Training  (NET) 

■  Independent  of  SORDAC;  Directly  communicate  with  CDR  USSOCOM 


UNCLASSIFIED 


J8-0 
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Operationally  Effective 


UNCLASSIFIED 


■  The  overall  degree  of  mission  accomplishment  of  a  system  when 
used  by  representative  personnel  in  the  environment  planned  or 
expected  for  operational  employment  of  the  system  considering 
organization,  doctrine,  survivability,  vulnerability  and  threat. 


■  SOCOM  OT&E  Focus 

>  Linked  to  Mission  Accomplishment 

>  What  Operational  Capabilities  are  Critical  to  Mission  Accomplishment 

>  Test  Environment  Adequate 


Meets  Technical  and  Operational  Performance  Requirements 


UNCLASSIFIED 


J8-0 


8 


Operationally  Suitable 


UNCLASSIFIED 


■  The  degree  to  which  a  system  can  be  satisfactorily  placed  in  field 
use,  with  consideration  given  to  availability,  compatibility, 
transportability,  interoperability,  reliability,  wartime  usage  rates, 
maintainability,  safety,  human  factors,  manpower,  supportability, 
logistics  supportability,  documentation,  and  training  requirements 


■  SOCOM  OT&E  Focus 

>  Linked  to  Mission  Accomplishment 

>  What  Operational  Capabilities  are  Critical  to  Mission  Accomplishment 

>  Test  Environment  Adequate 

Compatible  With  U.S./DOD/Service  Processes  and  Facilities 


UNCLASSIFIED 


J8-0 
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Operationally  Safe 


UNCLASSIFIED 


■  Identify  Hazards  and  Eliminate  or  Mitigate  those  Hazards  to  an 
Acceptable  Level 


■  Concurrently  Satisfy  the  Technical  Performance  Parameters  in  CPD, 
CDD,  or  equivalent 


■  USSOCOM  System  Safety  Risk  Assessment 


■  Safety  Certification  Authorities 


Safe  to  Use,  Handle,  Transport,  Store  and  Demilitarize 


UNCLASSIFIED 


J8-0 
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Documentation-  F&DR 


UNCLASSIFIED 


Fielding  and  Deployment  Release  (F&DR)  is  the  Final 

Documentation  Required  to  Certify  that  a  SOCOM  System  is 
Ready  for  Fielding: 

>  J4  Document  (Logistics) 

>  Certification  to  the  Milestone  Decision  Authority  that  all  Issues 
Identified  in  SPC  are  Satisfied  -  Addressing  Primarily  the 
Effectiveness,  Suitability,  Reliability,  Safety. 

>  Can  be  a  Combined  SPC/F&DR  Depending  upon  how  System 
is  Managed 


UNCLASSIFIED 


J8-0 
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UNCLASSIFIED 

SOCOM  T&E  Environment 


■  High  Risk  Environment 

>  Delayed  Point  of  Knowledge 

>  Further  delayed  Point  of  Influence 

■  Medium  Risk  Environment 

>  Delayed  but  synchronized  Point  of  Knowledge  and  Point  of 
Influence 

■  Low  Risk  Environment:  Integrated  DT/OT 

>  Up-front/  Early  Point  of  Knowledge  and  Point  of  Influence 


UNCLASSIFIED 


J8-0 


12 


Program  Investment 


High  Risk  Environment 


UNCLASSIFIED 


Point  of  Influence 


Just-in-Time  OE/OS 


J8-0 


UNCLASSIFIED 


13 


Program  Investment 


UNCLASSIFIED 


Medium  Risk  Environment 


Just-in -Time  OE/OS 
Plus 

Executability  Insight 


UNCLASSIFIED 


J8-0 
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Program  Investment 


UNCLASSIFIED 


Low  Risk  Environment- 
Integrated  T&E 


UNCLASSIFIED 


J8-0 
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JMA 


T&E  Implementation 


Requirements 


4 


Assessments 


SORDAC  OT&E 


Programmers 


Statutory 
10  USC  167,  2430,137,  87 


Regulatory 

OMB,  DOD5000,  CJCSI  ... 


D 

R 


UNCLASSIFIED 


USSOCOM  Implementation 
70-1,71-4,  71-5  ... 


Warfighter 


Repeat 


UNCLASSIFIED 


J8-0 
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UNCLASSIFIED 


Example: 

MK  20  MOD  0  Sniper  Support 


(SSR) 


UNCLASSIFIED 


J8-0 
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UNCLASSIFIED 


Example: 

MK  20  MOD  0  Sniper  Support 


(SSR) 


■  Originally  competed  as  “SV”  in  CAR  Solicitation 

■  User  Assessment  (UA) 

>  Conducted  at  Camp  Billy  Machen,  CA:  22-27  Feb  2009 

>  9  Test  Operators 

>  NSW,  USASOC  &  MARSOC 

■  Developmental  Test  (DT) 

>  Conducted  (Endurance)  at  FNMI,  Columbia,  SC:  8-11  Nov  2009 

>  NSWC  Crane  Personnel  Participated 

■  Operational  Test  &  Evaluation  (OT&E) 

>  Conducted  at  Camp  Billy  Machen,  CA:  6-18  Dec  2009 

>  10  Test  Operators 

>  NSW,  USASOC  &  MARSOC 

■  Achieved  Joint  Safety  Approval  on  1  June  2010 

■  Achieved  F&DR  on  18  Aug  10 

■  Note:  Core  Operators  continued  throughout  this  process 

J8-0 


UNCLASSIFIED 
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UNCLASSIFIED 


Example: 

SOF  Tactical  Combat  Casualty  Care 


Transport  Set  Adapt 

SO  Land,  Sea,  &  Air 

Echelon  ot  Care  1+ 

Mobility  Set  Rigid 
Flexible  Litters  & 

Litter  Aids 

1 _ 

Operator 

Platforms  for 

CASEVAC 

Kits 

Point  of 
Wounding 


Casualty 

Collection 

Theater  CASEVAC 

"  ^  «  •  »  * 

Point 

♦ 

Pick  Up  Zone 

% 

SOF 

Medic 


Medic 

Kits 


Extraction  Set 

Locate  &  Gain 
Access  to  Casualty 


Unit  Medical 
Officer 


Definitive  Care  Capabilities 


Echelon  1 
Plus  Care 


ICCC  SOF 
rauma  Mgmt 


Extraction  Mobility 

Locate  &  Gain  Access  Litters  &  Aids 


Sustainment 

Stabilization 


Transport 

Land,  Sea,  Air 


*  Water  & 

*  Transport 

*  Vitals 

*  Blood  Loss 

Night 

*  Various 

*  Cardiac 

*  Inspired  Oxy 

*Advanced 

*  High  Angle 

Vehicles 

*  Blood/Fluid 

Airway 

*  Pry  &  Cut 

*  Hypothermia 
*Secure 

Equipment 

UNCLASSIFIED 
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Example: 

SOF  Tactical  Combat  Casualty  Care  (TCCC) 


■  PMO  involved  with  J8-0  early,  during  establishment  of  Acquisition  Program 
Baseline,  to  ensure  OT  adequately  included  in  plan 

>  J8-0  test  input  reflected  in  Section  M  of  Contract  Solicitation 

■  Source  Selection  Evaluation  Board  (SSEB)  conducted  7-18  Jun  2010 

>  9  members  with  user  reps  from:  NSW,  AFSOC,  MARSOC  &  USASOC 

■  Developmental  Testing  (DT) 

>  Received  DTC  Safety  Release  ISO  Operational  Testing:  Nov  2010 

>  USAARL:  Air  Worthiness  and  environmental  (ongoing) 

>  USAAMA:  Medical  Sustainability  (ongoing) 

■  Operational  Test  &  Evaluation  (OT&E) 

>  Conducted  product  demonstration  Phase  1 :  Fort  Carson,  CO  -  Jan/  Feb  1 1 

>  15  Test  Operators:  NSW,  AFSOC,  MARSOC  &  USASOC 

>  Conduct  product  demonstration  Phase  2:  Crawfordsville,  AR  -  Mar 7  Apr  1 1 

>  16  Test  Operators:  NSW,  AFSOC,  MARSOC  &  USASOC 


UNCLASSIFIED 


J8-0 
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Conclusion- 
USSOCOM  Acquisition  Challenges 


UNCLASSIFIED 


■  Diversity  of  Platforms,  Users,  Systems,  Operating 
Environments,  and  Program  Management 
Structures 

■  Mandate  to  Rapidly  Respond  to  Emergent 
Warfighter  Requirements 

>  80%  Solutions 

>  Take  Risk  and  Manage  It 

>  Many  SOCOM  Systems  Installed  in  Platforms  Owned  and 
Developed  by  Others  (Services,  MILDEPS) 

>  Contractor-off-the-shelf  /  Non-developmental  Items 


UNCLASSIFIED 


J8-0 
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Our  Reason  for  Bein 
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